[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156314#comment-14156314
]
Hudson commented on HDFS-6931:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/698/])
HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup.
(Arpit Agarwal) (arp: rev c92837aeab5188f6171d4016f91b3b4936a66beb)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: 3.0.0
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{current/lazyPersist/}}
directory to {{current/finalized}}. Duplicate replicas of the same block
should be deleted from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156392#comment-14156392
]
Hudson commented on HDFS-6931:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/])
HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup.
(Arpit Agarwal) (arp: rev c92837aeab5188f6171d4016f91b3b4936a66beb)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: 3.0.0
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{current/lazyPersist/}}
directory to {{current/finalized}}. Duplicate replicas of the same block
should be deleted from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156508#comment-14156508
]
Hudson commented on HDFS-6931:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/])
HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup.
(Arpit Agarwal) (arp: rev c92837aeab5188f6171d4016f91b3b4936a66beb)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: 3.0.0
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{current/lazyPersist/}}
directory to {{current/finalized}}. Duplicate replicas of the same block
should be deleted from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155005#comment-14155005
]
Hudson commented on HDFS-6931:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6163/])
HDFS-6931. Move lazily persisted replicas to finalized directory on DN startup.
(Arpit Agarwal) (arp: rev c92837aeab5188f6171d4016f91b3b4936a66beb)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestLazyPersistFiles.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: HDFS-6581
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{current/lazyPersist/}}
directory to {{current/finalized}}. Duplicate replicas of the same block
should be deleted from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114531#comment-14114531
]
Xiaoyu Yao commented on HDFS-6931:
--
+1
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: HDFS-6581
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{current/lazyPersist/}}
directory to {{current/finalized}}. Duplicate replicas of the same block
should be deleted from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114882#comment-14114882
]
Jitendra Nath Pandey commented on HDFS-6931:
+1
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: HDFS-6581
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{current/lazyPersist/}}
directory to {{current/finalized}}. Duplicate replicas of the same block
should be deleted from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
[
https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113329#comment-14113329
]
Arpit Agarwal commented on HDFS-6931:
-
On restart each volume will be scanned and replicas under {{lazyPersist/}} will
be moved to their corresponding locations under {{finalized/}}.
We may end up with two replicas of the same block on different volumes, so we
use the following scheme to decide which replica to keep. See
{{BlockPoolSlice.resolveDuplicateReplicas}}.
{code}
* Given two replicas, decide which one to keep. The preference is as
* follows:
* 1. Prefer the replica with the higher generation stamp.
* 2. If generation stamps are equal, prefer the replica with the
* larger on-disk length.
* 3. If on-disk length is the same, prefer the replica on persistent
* storage volume.
* 4. All other factors being equal, keep replica1.
*
* The other replica is removed from the volumeMap and is deleted from
* its storage volume.
{code}
Also added test cases.
Move lazily persisted replicas to finalized directory on DN startup
---
Key: HDFS-6931
URL: https://issues.apache.org/jira/browse/HDFS-6931
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Fix For: HDFS-6581
Attachments: HDFS-6931.01.patch
On restart the DN should move replicas from the {{lazyPersist}} directory to
{{finalized}}, and evict/unlink replicas of the same block from RAM disk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
