[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14805194#comment-14805194 ] Kanaka Kumar Avvaru commented on HDFS-6955: --- [~vinayrpet], attached patch for branch-2 based on v9. > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14846269#comment-14846269 ] Hudson commented on HDFS-6955: -- FAILURE: Integrated in Hadoop-trunk-Commit #8477 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8477/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14840232#comment-14840232 ] Kanaka Kumar Avvaru commented on HDFS-6955: --- Thanks for the reviews & commit [~vinayrpet]. Thanks for the reviews [~arpitagarwal]. > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14847328#comment-14847328
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 43s | Pre-patch branch-2 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 6m 3s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 23s | The applied patch generated 2
new checkstyle issues (total was 200, now 199). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 13s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 1m 33s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 189m 48s | Tests failed in hadoop-hdfs. |
| | | 231m 11s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12757236/%0AHDFS-6955-branch-2.09.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | branch-2 / d94c092 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12539/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12539/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12539/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12539/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Fix For: 2.8.0
>
> Attachments:
HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch,
HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch,
HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14847362#comment-14847362 ] Hudson commented on HDFS-6955: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #407 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/407/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14847332#comment-14847332 ] Hudson commented on HDFS-6955: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1148 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1148/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875695#comment-14875695 ] Hudson commented on HDFS-6955: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2354 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2354/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14852763#comment-14852763 ] Hudson commented on HDFS-6955: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #414 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/414/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875848#comment-14875848 ] Hudson commented on HDFS-6955: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #390 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/390/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14875906#comment-14875906 ] Hudson commented on HDFS-6955: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2329 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2329/]) HDFS-6955. DN should reserve disk space for a full block when creating tmp files (Contributed by Kanaka Kumar Avvaru) (vinayakumarb: rev 92c1af1646b1d91a2ab7821e4f7d450e3b6e10bb) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestRbwSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestSpaceReservation.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Fix For: 2.8.0 > > Attachments: HDFS-6955-branch-2.09.patch, HDFS-6955-01.patch, HDFS-6955-02.patch, HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14804928#comment-14804928 ] Vinayakumar B commented on HDFS-6955: - Seems there is a little conflict in committing this to branch-2. [~kanaka], Please post a branch-2 version of latest patch as well. -Thanks > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, > HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747025#comment-14747025 ] Kanaka Kumar Avvaru commented on HDFS-6955: --- deleted and uploaded v9 again to trigger jenkins build. > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, > HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747337#comment-14747337 ] Kanaka Kumar Avvaru commented on HDFS-6955: --- Test failures seems not related to the changes in this patch. > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, > HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14747283#comment-14747283
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 20s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 59s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 10s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 11s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 117m 52s | Tests failed in hadoop-hdfs. |
| | | 161m 12s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory |
| | hadoop.hdfs.server.namenode.ha.TestGetGroupsWithHA |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| | hadoop.hdfs.server.namenode.ha.TestStandbyBlockManagement |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12756183/HDFS-6955-09.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 452079a |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12473/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12473/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12473/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch, HDFS-6955-09.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741256#comment-14741256
]
Arpit Agarwal commented on HDFS-6955:
-
Thanks for updating the patch. A few comments.
# Nitpick: Please add spaces around operators {{replicaInfo!=null}}
# It doesn't look like there is a new test case for temporary reservation. We
should have test case for both full and interrupted transfer.
# {{FsVolumeImpl#recentReserved}} looks unused. Let's remove it.
# I didn't recall the state machine, do temporary blocks get finalized?
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741365#comment-14741365
]
Kanaka Kumar Avvaru commented on HDFS-6955:
---
Thanks for the review [~arpitagarwal]
1. Ok. fine
2. New tests are added for full success and volume errors. Please refer
{{TestSpaceReservation#testTmpSpaceReserve()}}
3. it is to be used in tests for verification for recent reserved space for
last call to assert the actual block length instead of default block size.
4. Yes, based on the code flow, block finalization is happening for
re-replicated block also
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741374#comment-14741374
]
Kanaka Kumar Avvaru commented on HDFS-6955:
---
patch v8 address {{TestSpaceReservation}} test failure in last build and
comment from arpit for spaces around operators {{replicaInfo!=null}}
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741694#comment-14741694
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 40s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 3
new checkstyle issues (total was 197, now 198). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 19s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 149m 22s | Tests failed in hadoop-hdfs. |
| | | 194m 35s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.namenode.TestMetadataVersionOutput |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory |
| | hadoop.hdfs.server.namenode.TestMetaSave |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter |
| Timed out tests | org.apache.hadoop.hdfs.server.datanode.TestDeleteBlockPool |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755469/HDFS-6955-08.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ea4bb27 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12402/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12402/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12402/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12402/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch, HDFS-6955-08.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14740898#comment-14740898
]
Kanaka Kumar Avvaru commented on HDFS-6955:
---
Thanks for review [~arpitagarwal]. Thanks for tracking up [~vinayrpet].
Combining both reservations to one counter is a good suggestion. Updated patch
for the same.
{{reservedSpaceForRBW}} is recently added in JMX and not delivered in any
release yet. So, we can safely change this also to {{reservedSpaceForReplicas}}
Please review new patch.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14741159#comment-14741159
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 20m 1s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 9m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 51s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 27s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 45s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 57s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 44s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 7s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 48s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 160m 27s | Tests failed in hadoop-hdfs. |
| | | 213m 4s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks |
| | hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks |
| | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation |
| Timed out tests |
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| |
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestScrLazyPersistFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12755401/HDFS-6955-07.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 486d5cb |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/12398/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12398/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12398/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12398/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch,
> HDFS-6955-06.patch, HDFS-6955-07.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738657#comment-14738657 ] Vinayakumar B commented on HDFS-6955: - thanks [~kanaka] for the update. +1, Changes looks good to me in latest patch. [~arpitagarwal], do you also want to take a look? Will wait for a day before commit. -Thanks > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739300#comment-14739300 ] Arpit Agarwal commented on HDFS-6955: - Hi [~vinayrpet], good point. Perhaps we can just document that the metric tracks space reserved for both RBW and tmp replicas. A single counter would be easier to maintain. > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739306#comment-14739306 ] Vinayakumar B commented on HDFS-6955: - bq. Hi Vinayakumar B, good point. Perhaps we can just document that the metric tracks space reserved for both RBW and tmp replicas. A single counter would be easier to maintain. You mean, no need to rename the reservedForRbw counter to reserved space? just use for tmp files too as is? > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739312#comment-14739312 ] Arpit Agarwal commented on HDFS-6955: - We can rename the internal counter to reservedForReplicas or something similar but leave the metric as it is for compatibility. What do you think? > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739318#comment-14739318 ] Vinayakumar B commented on HDFS-6955: - Yes, that would work. Thank you. [~kanaka], can you please update based on above suggestion by Arpit? > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739253#comment-14739253 ] Arpit Agarwal commented on HDFS-6955: - Thanks for fixing this [~kanaka]. Do we need the separate counter for tmp or can we just rename the reservedForRbw counter to reserved space? > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14739273#comment-14739273 ] Vinayakumar B commented on HDFS-6955: - bq. Do we need the separate counter for tmp or can we just rename the reservedForRbw counter to reserved space? Thats the good point [~arpitagarwal]. We could combine it, but 'reservedSpaceForRBW' is exposed as VolumeInfo metric. If we combine it, then metric will have incompatible change. What do you say, is it okay to combine despite incompatible change in metric? > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14736316#comment-14736316
]
Vinayakumar B commented on HDFS-6955:
-
bq. 1. Reserved space is not released unless we call cleanupBlock(). Yes its
good point to verify double release . I have updated the test case to assert
this case. Please correct me if still you feel some other call is clearing
space.
Thanks for pointing out that, {{bytesReserved}} is resetting to 0 in
{{releaseAllBytesReserved}}, then double calls wont harm.
I think the following change need not required and this will not reset the
value of bytesReserved in replicainfo. Instead, calling
{{replicaInfo.releaseAllBytesReserved();}} will make sure everything reset. It
would be better to unify this call everywhere.
{code}-
+ FsVolumeImpl v = (FsVolumeImpl)replicaInfo.getVolume();
+ v.releaseReservedSpaceTmp(replicaInfo.getBytesReserved());
+ {code}
One more place {{replicaInfo.releaseAllBytesReserved();}} call required is the
{{BlockReceiver}}'s constructor on exception. This will make sure reserved
space is cleared even after some other exception during creation of block (
either rbw or tmp).
It would be better if more tests can be added to verify these corner cases.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737813#comment-14737813
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 7s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 2
new checkstyle issues (total was 215, now 215). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 169m 29s | Tests failed in hadoop-hdfs. |
| | | 214m 30s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.web.TestWebHDFSOAuth2 |
| Timed out tests | org.apache.hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12754981/HDFS-6955-06.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4014ce5 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12360/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12360/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12360/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12360/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14738223#comment-14738223 ] Kanaka Kumar Avvaru commented on HDFS-6955: --- Test failure looks not related to the changes done in this patch. Check style issues can be ignored as file length was big before the patch and accessor methods are not required for "reservedSpaceForTmp" > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14737449#comment-14737449
]
Kanaka Kumar Avvaru commented on HDFS-6955:
---
Thanks for the review [~vinayrpet].
Its a good suggestion to unify the release through
{{replicaInfo.releaseAllBytesReserved()}} as this call is safe for multiple
invocations also. Actually there is a leak exists if exception occurs just
after {{createRBWFile}} (Difficult to write test case so tried through debug &
caused exceptions through code inspection)
I have updated patch to remove the clearing logic in {{cleanupBlock()}} and use
{{releaseAllBytesReserved()}} in {{BlockReceiver}} constructor on exception
case which can handle both RBW & Tmp spaces clearing.
I have done small changes in test code to ensure release additional space is
not happening.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch, HDFS-6955-06.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14733165#comment-14733165
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 24s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 5 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 30s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 186m 53s | Tests passed in hadoop-hdfs.
|
| | | 228m 44s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12754404/HDFS-6955-05.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 9b68577 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12323/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12323/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12323/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14732518#comment-14732518
]
Kanaka Kumar Avvaru commented on HDFS-6955:
---
Thanks for the review [~vinayrpet]
1. Reserved space is not released unless we call {{cleanupBlock()}}. Yes its
good point to verify double release . I have updated the test case to assert
this case. Please correct me if still you feel some other call is clearing
space.
2. Thanks for pointing the name correction. I have renamed in latest version of
patch (v5)
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch, HDFS-6955-05.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14730558#comment-14730558
]
Vinayakumar B commented on HDFS-6955:
-
Thanks [~kanaka] for taking up the issue.
Patch looks almost good.
Some nits.
1. I think releasing reservation in {{cleanupBlock()}} not required. As by the
time this method is called in {{BlockReceiver}}, already reserved space will be
released. So this would release again.
2. Since Now reservation is for both RBW and Temp files, renaming
{{TestRbwSpaceReservation.java}} to something like
{{TestSpaceReservation.java}} would be a good idea IMO. If you feel existing is
fine, then no issues.
[~arpitagarwal], Do you also want to take a look at the patch.?
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: Kanaka Kumar Avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724891#comment-14724891 ] kanaka kumar avvaru commented on HDFS-6955: --- Test failure looks not related to the changes done in this patch. Check style issues can be ignored as file length was big before the patch and accessor methods are not required for "reservedSpaceForTmp" > DN should reserve disk space for a full block when creating tmp files > - > > Key: HDFS-6955 > URL: https://issues.apache.org/jira/browse/HDFS-6955 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.5.0 >Reporter: Arpit Agarwal >Assignee: kanaka kumar avvaru > Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch, > HDFS-6955-03.patch, HDFS-6955-04.patch > > > HDFS-6898 is introducing disk space reservation for RBW files to avoid > running out of disk space midway through block creation. > This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724106#comment-14724106
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 2s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 54s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 7s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 20s | The applied patch generated 2
new checkstyle issues (total was 155, now 155). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 37s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 16s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 70m 1s | Tests failed in hadoop-hdfs. |
| | | 115m 46s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| Timed out tests | org.apache.hadoop.hdfs.TestDatanodeDeath |
| | org.apache.hadoop.hdfs.TestDecommission |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12753357/HDFS-6955-04.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / caa04de |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12218/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12218/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12218/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12218/console |
This message was automatically generated.
> DN should reserve disk space for a full block when creating tmp files
> -
>
> Key: HDFS-6955
> URL: https://issues.apache.org/jira/browse/HDFS-6955
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: Arpit Agarwal
>Assignee: kanaka kumar avvaru
> Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
> HDFS-6955-03.patch, HDFS-6955-04.patch
>
>
> HDFS-6898 is introducing disk space reservation for RBW files to avoid
> running out of disk space midway through block creation.
> This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14712167#comment-14712167
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 17m 43s | Pre-patch trunk has 4 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 21s | The applied patch generated 3
new checkstyle issues (total was 153, now 154). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 30s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 10s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 164m 1s | Tests failed in hadoop-hdfs. |
| | | 208m 53s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.web.TestWebHDFS |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752303/HDFS-6955-03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12126/console |
This message was automatically generated.
DN should reserve disk space for a full block when creating tmp files
-
Key: HDFS-6955
URL: https://issues.apache.org/jira/browse/HDFS-6955
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: kanaka kumar avvaru
Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch,
HDFS-6955-03.patch
HDFS-6898 is introducing disk space reservation for RBW files to avoid
running out of disk space midway through block creation.
This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14700530#comment-14700530
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 50s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 8m 2s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 54s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 3
new checkstyle issues (total was 154, now 155). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 24s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 9s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 176m 49s | Tests failed in hadoop-hdfs. |
| | | 222m 14s | |
\\
\\
|| Reason || Tests ||
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750865/HDFS-6955-02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c77bd6a |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12013/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12013/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12013/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12013/console |
This message was automatically generated.
DN should reserve disk space for a full block when creating tmp files
-
Key: HDFS-6955
URL: https://issues.apache.org/jira/browse/HDFS-6955
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: kanaka kumar avvaru
Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch
HDFS-6898 is introducing disk space reservation for RBW files to avoid
running out of disk space midway through block creation.
This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14699926#comment-14699926
]
kanaka kumar avvaru commented on HDFS-6955:
---
Updated patch for white space errors and a checkstyle issue
{{FsVolumeImpl.java:462:69: 'reserved' hides a field.}}.
DN should reserve disk space for a full block when creating tmp files
-
Key: HDFS-6955
URL: https://issues.apache.org/jira/browse/HDFS-6955
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: kanaka kumar avvaru
Attachments: HDFS-6955-01.patch, HDFS-6955-02.patch
HDFS-6898 is introducing disk space reservation for RBW files to avoid
running out of disk space midway through block creation.
This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[
https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14697397#comment-14697397
]
Hadoop QA commented on HDFS-6955:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 23s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 22s | The applied patch generated 3
new checkstyle issues (total was 154, now 155). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 41s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 55s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 146m 6s | Tests failed in hadoop-hdfs. |
| | | 191m 58s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| Timed out tests |
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureOfSharedDir |
| | org.apache.hadoop.hdfs.server.namenode.TestDeadDatanode |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750505/HDFS-6955-01.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 84bf712 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11997/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11997/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11997/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11997/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11997/console |
This message was automatically generated.
DN should reserve disk space for a full block when creating tmp files
-
Key: HDFS-6955
URL: https://issues.apache.org/jira/browse/HDFS-6955
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.5.0
Reporter: Arpit Agarwal
Assignee: kanaka kumar avvaru
Attachments: HDFS-6955-01.patch
HDFS-6898 is introducing disk space reservation for RBW files to avoid
running out of disk space midway through block creation.
This Jira is to introduce similar reservation for tmp files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593411#comment-14593411 ] kanaka kumar avvaru commented on HDFS-6955: --- Hi [~arpitagarwal], I would like to work on this JIRA. Please share any inputs if you have already thought of. DN should reserve disk space for a full block when creating tmp files - Key: HDFS-6955 URL: https://issues.apache.org/jira/browse/HDFS-6955 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Arpit Agarwal HDFS-6898 is introducing disk space reservation for RBW files to avoid running out of disk space midway through block creation. This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6955) DN should reserve disk space for a full block when creating tmp files
[ https://issues.apache.org/jira/browse/HDFS-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14593834#comment-14593834 ] Arpit Agarwal commented on HDFS-6955: - Hi [~kanaka], no I haven't thought much about this one yet. The idea is to reserve space for blocks being re-replicated. DN should reserve disk space for a full block when creating tmp files - Key: HDFS-6955 URL: https://issues.apache.org/jira/browse/HDFS-6955 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Arpit Agarwal HDFS-6898 is introducing disk space reservation for RBW files to avoid running out of disk space midway through block creation. This Jira is to introduce similar reservation for tmp files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
