[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003481#comment-14003481 ] Hudson commented on HDFS-6325: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1780 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1780/]) HDFS-6325. Append should fail if the last block has insufficient number of replicas (Keith Pak via cos) (cos: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1595744) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Fix For: 2.4.1 > > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003304#comment-14003304 ] Hudson commented on HDFS-6325: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1754 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1754/]) HDFS-6325. Append should fail if the last block has insufficient number of replicas (Keith Pak via cos) (cos: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1595744) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Fix For: 2.4.1 > > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003269#comment-14003269 ] Hudson commented on HDFS-6325: -- FAILURE: Integrated in Hadoop-Yarn-trunk #562 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/562/]) HDFS-6325. Append should fail if the last block has insufficient number of replicas (Keith Pak via cos) (cos: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1595744) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Fix For: 2.4.1 > > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14002951#comment-14002951 ] Hudson commented on HDFS-6325: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5606 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5606/]) HDFS-6325. Append should fail if the last block has insufficient number of replicas (Keith Pak via cos) (cos: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1595744) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend4.java > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Fix For: 2.4.1 > > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13998517#comment-13998517
]
Konstantin Shvachko commented on HDFS-6325:
---
This looks good. Just two final touches:
- In appendFileInternal() you can actually combine nested if statements into
one with three conditions.
- In testAppendInsufficientLocations() you should use {{LOG.info("message",
e);}} instead of {{+ e.getMessage()}}
The patch cleanly applies to branch-2 and branch-2.4.
It would be nice to have it in 2.4.1. If there are no there objections?
> Append should fail if the last block has insufficient number of replicas
>
>
> Key: HDFS-6325
> URL: https://issues.apache.org/jira/browse/HDFS-6325
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Konstantin Shvachko
>Assignee: Keith Pak
> Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch,
> HDFS-6325_test.patch, appendTest.patch
>
>
> Currently append() succeeds on a file with the last block that has no
> replicas. But the subsequent updatePipeline() fails as there are no replicas
> with the exception "Unable to retrieve blocks locations for last block". This
> leaves the file unclosed, and others can not do anything with it until its
> lease expires.
> The solution is to check replicas of the last block on the NameNode and fail
> during append() rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two
> options:
> # min-replication: allow append if the last block is minimally replicated (1
> by default)
> # full-replication: allow append if the last block is fully replicated (3 by
> default)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999637#comment-13999637 ] Konstantin Shvachko commented on HDFS-6325: --- +1 looks good. > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999576#comment-13999576
]
Hadoop QA commented on HDFS-6325:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645053/HDFS-6325.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6913//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6913//console
This message is automatically generated.
> Append should fail if the last block has insufficient number of replicas
>
>
> Key: HDFS-6325
> URL: https://issues.apache.org/jira/browse/HDFS-6325
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Konstantin Shvachko
>Assignee: Keith Pak
> Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch,
> HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch
>
>
> Currently append() succeeds on a file with the last block that has no
> replicas. But the subsequent updatePipeline() fails as there are no replicas
> with the exception "Unable to retrieve blocks locations for last block". This
> leaves the file unclosed, and others can not do anything with it until its
> lease expires.
> The solution is to check replicas of the last block on the NameNode and fail
> during append() rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two
> options:
> # min-replication: allow append if the last block is minimally replicated (1
> by default)
> # full-replication: allow append if the last block is fully replicated (3 by
> default)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999576#comment-13999576
]
Hadoop QA commented on HDFS-6325:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645053/HDFS-6325.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6913//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6913//console
This message is automatically generated.
> Append should fail if the last block has insufficient number of replicas
>
>
> Key: HDFS-6325
> URL: https://issues.apache.org/jira/browse/HDFS-6325
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Konstantin Shvachko
>Assignee: Keith Pak
> Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch,
> HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch
>
>
> Currently append() succeeds on a file with the last block that has no
> replicas. But the subsequent updatePipeline() fails as there are no replicas
> with the exception "Unable to retrieve blocks locations for last block". This
> leaves the file unclosed, and others can not do anything with it until its
> lease expires.
> The solution is to check replicas of the last block on the NameNode and fail
> during append() rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two
> options:
> # min-replication: allow append if the last block is minimally replicated (1
> by default)
> # full-replication: allow append if the last block is fully replicated (3 by
> default)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995837#comment-13995837
]
Hadoop QA commented on HDFS-6325:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644461/HDFS-6325.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.TestFileAppend4
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6879//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6879//console
This message is automatically generated.
> Append should fail if the last block has insufficient number of replicas
>
>
> Key: HDFS-6325
> URL: https://issues.apache.org/jira/browse/HDFS-6325
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Konstantin Shvachko
>Assignee: Keith Pak
> Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch,
> appendTest.patch
>
>
> Currently append() succeeds on a file with the last block that has no
> replicas. But the subsequent updatePipeline() fails as there are no replicas
> with the exception "Unable to retrieve blocks locations for last block". This
> leaves the file unclosed, and others can not do anything with it until its
> lease expires.
> The solution is to check replicas of the last block on the NameNode and fail
> during append() rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two
> options:
> # min-replication: allow append if the last block is minimally replicated (1
> by default)
> # full-replication: allow append if the last block is fully replicated (3 by
> default)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[
https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996013#comment-13996013
]
Hadoop QA commented on HDFS-6325:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644527/HDFS-6325.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6885//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6885//console
This message is automatically generated.
> Append should fail if the last block has insufficient number of replicas
>
>
> Key: HDFS-6325
> URL: https://issues.apache.org/jira/browse/HDFS-6325
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.2.0
>Reporter: Konstantin Shvachko
>Assignee: Keith Pak
> Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch,
> HDFS-6325_test.patch, appendTest.patch
>
>
> Currently append() succeeds on a file with the last block that has no
> replicas. But the subsequent updatePipeline() fails as there are no replicas
> with the exception "Unable to retrieve blocks locations for last block". This
> leaves the file unclosed, and others can not do anything with it until its
> lease expires.
> The solution is to check replicas of the last block on the NameNode and fail
> during append() rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two
> options:
> # min-replication: allow append if the last block is minimally replicated (1
> by default)
> # full-replication: allow append if the last block is fully replicated (3 by
> default)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13995918#comment-13995918 ] Keith Pak commented on HDFS-6325: - testFileAppend4 failure: There may be an overlap of shutdown of testAppendInsufficientLocations and startup of testCompleteOtherLeaseHoldersFile. I changed testAppendInsufficientLocations to use the field variable "cluster" instead of creating my own as a local variable and also reduced the number of DNs from 6 to 4. Otherwise, I see that testCompleteOtherLeaseHoldersFile is progressing so I think a bigger timeout may be required as stated in this JIRA: https://issues.apache.org/jira/browse/HADOOP-8596. TestBalancerWithNodeGroup: This also seems to be an intermittent issue from https://issues.apache.org/jira/browse/HDFS-6250. > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, > HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994737#comment-13994737 ] Konstantin Shvachko commented on HDFS-6325: --- Here are some thoughts after experimenting and discussions. # We should still allow executing append if the number of live DNs on the cluster is less than file replication factor. This is to support cases with small clusters or replication being set intentionally high. # We should probably go for option (1) with min-replication. This has an advantage that there will be no change in semantics with current implementation. The disadvantage is that once file is opened for append the ongoing replications of the last block will fail, because of the genStamp change. # In the end our main goal is to make append fail earlier to prevent the file from being under construction while nobody can use it, which happens when the last block has 0 replicas. > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994738#comment-13994738 ] Konstantin Shvachko commented on HDFS-6325: --- Patch review comments: # Need to change full-replication to min-replication. # You should not check sufficient replication if the last block is already under construction. # JavaDoc comment for isSufficientlyReplicated() is not clear and the comment in the body is redundant. # The test looks good. I see a few long lines though. > Append should fail if the last block has insufficient number of replicas > > > Key: HDFS-6325 > URL: https://issues.apache.org/jira/browse/HDFS-6325 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 >Reporter: Konstantin Shvachko >Assignee: Keith Pak > Attachments: HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch > > > Currently append() succeeds on a file with the last block that has no > replicas. But the subsequent updatePipeline() fails as there are no replicas > with the exception "Unable to retrieve blocks locations for last block". This > leaves the file unclosed, and others can not do anything with it until its > lease expires. > The solution is to check replicas of the last block on the NameNode and fail > during append() rather than during updatePipeline(). > How many replicas should be present before NN allows to append? I see two > options: > # min-replication: allow append if the last block is minimally replicated (1 > by default) > # full-replication: allow append if the last block is fully replicated (3 by > default) -- This message was sent by Atlassian JIRA (v6.2#6252)
