[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156430#comment-14156430 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java Add unit test for evict/delete RAM_DISK block with open handle -- Key: HDFS-6990 URL: https://issues.apache.org/jira/browse/HDFS-6990 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 3.0.0 Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, HDFS-6990.3.patch This is to verify: * Evict RAM_DISK block with open handle should fall back to DISK. * Delete RAM_DISK block (persisted) with open handle should mark the block to be deleted upon handle close. Simply open handle to file in DFS name space won't work as expected. We need a local FS file handle to the block file. The only meaningful case is for Short Circuit Read. This JIRA is to validate/enable the two cases with SCR enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14156546#comment-14156546 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt Add unit test for evict/delete RAM_DISK block with open handle -- Key: HDFS-6990 URL: https://issues.apache.org/jira/browse/HDFS-6990 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: 3.0.0 Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, HDFS-6990.3.patch This is to verify: * Evict RAM_DISK block with open handle should fall back to DISK. * Delete RAM_DISK block (persisted) with open handle should mark the block to be deleted upon handle close. Simply open handle to file in DFS name space won't work as expected. We need a local FS file handle to the block file. The only meaningful case is for Short Circuit Read. This JIRA is to validate/enable the two cases with SCR enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14155024#comment-14155024 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6163/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java Add unit test for evict/delete RAM_DISK block with open handle -- Key: HDFS-6990 URL: https://issues.apache.org/jira/browse/HDFS-6990 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Fix For: HDFS-6581 Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, HDFS-6990.3.patch This is to verify: * Evict RAM_DISK block with open handle should fall back to DISK. * Delete RAM_DISK block (persisted) with open handle should mark the block to be deleted upon handle close. Simply open handle to file in DFS name space won't work as expected. We need a local FS file handle to the block file. The only meaningful case is for Short Circuit Read. This JIRA is to validate/enable the two cases with SCR enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144132#comment-14144132 ] Arpit Agarwal commented on HDFS-6990: - +1 for the patch. Thanks for addressing all the rounds of feedback [~xyao]. I will commit it shortly. Add unit test for evict/delete RAM_DISK block with open handle -- Key: HDFS-6990 URL: https://issues.apache.org/jira/browse/HDFS-6990 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, HDFS-6990.3.patch This is to verify: * Evict RAM_DISK block with open handle should fall back to DISK. * Delete RAM_DISK block (persisted) with open handle should mark the block to be deleted upon handle close. Simply open handle to file in DFS name space won't work as expected. We need a local FS file handle to the block file. The only meaningful case is for Short Circuit Read. This JIRA is to validate/enable the two cases with SCR enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[
https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144136#comment-14144136
]
Arpit Agarwal commented on HDFS-6990:
-
FTR, here is the offline feedback I had on this patch:
{code}
+ // Ensure path1 is still readable from the open SCR handle.
+ fis.read(fis.getPos(), buf, 0, BUFFER_LENGTH);
+ assertThat(verifyReadRandomFile(path1, BLOCK_SIZE, SEED), is(true));
{code}
This is not reading from the same handle as before. We should read from the fis
handle we opened earlier, and then also check the short circuit read counters
to ensure the read was short-circuited.
These tests should be skipped on Windows or when native IO is not enabled. You
could add something like:
{code}
assumeTrue(NativeCodeLoader.isNativeCodeLoaded() !Path.WINDOWS);
{code}
See Chris's recent fix to HDFS-7110.
Also it is worth noting (maybe in a comment) we are not really testing SCR from
RAM_DISK, since the test fakes RAM_DISK on physical disk.
+1 with these updates.
Add unit test for evict/delete RAM_DISK block with open handle
--
Key: HDFS-6990
URL: https://issues.apache.org/jira/browse/HDFS-6990
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch,
HDFS-6990.3.patch
This is to verify:
* Evict RAM_DISK block with open handle should fall back to DISK.
* Delete RAM_DISK block (persisted) with open handle should mark the block to
be deleted upon handle close.
Simply open handle to file in DFS name space won't work as expected. We need
a local FS file handle to the block file. The only meaningful case is for
Short Circuit Read. This JIRA is to validate/enable the two cases with SCR
enabled MiniDFSCluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
