[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-6325: - Resolution: Fixed Release Note: I have committed the fix to the trunk, branch-2, and branch-2.4 respectively. Thanks Keith! Status: Resolved (was: Patch Available) Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Fix For: 2.4.1 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-6325: - Fix Version/s: 2.4.1 Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Fix For: 2.4.1 Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-6325: -- Target Version/s: 2.4.1 (was: 2.5.0) Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch Attached patch. Addressed the above comments and passed the TestFileAppend* tests locally. Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch 1. Fixed isSufficientlyReplicated and comments for minimum replication check. 2. Added a isComplete check so we dont check underconstruction files in FSNamesystem. 3. Fixed lines in test. Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: (was: HDFS-6325.patch) Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Status: Patch Available (was: In Progress) Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325.patch, HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Keith Pak updated HDFS-6325: Attachment: HDFS-6325.patch Here is a patch with a proposed fix to introduce BlockManager#isSufficientlyReplicated and a unit test. -This will check that the block is replicated to at least the min of number of live DNs and the replication factor of the file. The unit test will 1. start 6 DNs 2. create a file with a rep factor of 3 3. Kill the DNs with the block locations of that file. 4. run append and ensure failure 5. ensure the file is not opened afterwards. Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Assignee: Keith Pak Attachments: HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-6325: -- Summary: Append should fail if the last block has insufficient number of replicas (was: Append should fail if the last block has unsufficient number of replicas) Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
[ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-6325: --- Attachment: appendTest.patch Attaching patch to reproduce issue. This can reproduced on a MiniDFSCluster of 1 NameNode and 1 DataNode. Once the environment is in the state of NameNode having zero block locations of some file X and out of SafeMode the following events will happen: The first append to X will fail with 'unable to retrieve last block of file X'. Subsequent appends will fail with AlreadyBeingCreatedException until lease recovery occurs. Append should fail if the last block has insufficient number of replicas Key: HDFS-6325 URL: https://issues.apache.org/jira/browse/HDFS-6325 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Konstantin Shvachko Attachments: appendTest.patch Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception Unable to retrieve blocks locations for last block. This leaves the file unclosed, and others can not do anything with it until its lease expires. The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline(). How many replicas should be present before NN allows to append? I see two options: # min-replication: allow append if the last block is minimally replicated (1 by default) # full-replication: allow append if the last block is fully replicated (3 by default) -- This message was sent by Atlassian JIRA (v6.2#6252)
