[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156546#comment-14156546 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1914 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1914/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt > Add unit test for evict/delete RAM_DISK block with open handle > -- > > Key: HDFS-6990 > URL: https://issues.apache.org/jira/browse/HDFS-6990 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 3.0.0 > > Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, > HDFS-6990.3.patch > > > This is to verify: > * Evict RAM_DISK block with open handle should fall back to DISK. > * Delete RAM_DISK block (persisted) with open handle should mark the block to > be deleted upon handle close. > Simply open handle to file in DFS name space won't work as expected. We need > a local FS file handle to the block file. The only meaningful case is for > Short Circuit Read. This JIRA is to validate/enable the two cases with SCR > enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156430#comment-14156430 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1889 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1889/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java > Add unit test for evict/delete RAM_DISK block with open handle > -- > > Key: HDFS-6990 > URL: https://issues.apache.org/jira/browse/HDFS-6990 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 3.0.0 > > Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, > HDFS-6990.3.patch > > > This is to verify: > * Evict RAM_DISK block with open handle should fall back to DISK. > * Delete RAM_DISK block (persisted) with open handle should mark the block to > be deleted upon handle close. > Simply open handle to file in DFS name space won't work as expected. We need > a local FS file handle to the block file. The only meaningful case is for > Short Circuit Read. This JIRA is to validate/enable the two cases with SCR > enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156352#comment-14156352 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-Yarn-trunk #698 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/698/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java > Add unit test for evict/delete RAM_DISK block with open handle > -- > > Key: HDFS-6990 > URL: https://issues.apache.org/jira/browse/HDFS-6990 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: 3.0.0 > > Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, > HDFS-6990.3.patch > > > This is to verify: > * Evict RAM_DISK block with open handle should fall back to DISK. > * Delete RAM_DISK block (persisted) with open handle should mark the block to > be deleted upon handle close. > Simply open handle to file in DFS name space won't work as expected. We need > a local FS file handle to the block file. The only meaningful case is for > Short Circuit Read. This JIRA is to validate/enable the two cases with SCR > enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155024#comment-14155024 ] Hudson commented on HDFS-6990: -- FAILURE: Integrated in Hadoop-trunk-Commit #6163 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6163/]) HDFS-6990. Add unit test for evict/delete RAM_DISK block with open handle. (Contributed by Xiaoyu Yao) (arp: rev 8b139b0800b2724178d5c155842588e9593a939f) * hadoop-hdfs-project/hadoop-hdfs/CHANGES-HDFS-6581.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestScrLazyPersistFiles.java > Add unit test for evict/delete RAM_DISK block with open handle > -- > > Key: HDFS-6990 > URL: https://issues.apache.org/jira/browse/HDFS-6990 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Fix For: HDFS-6581 > > Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, > HDFS-6990.3.patch > > > This is to verify: > * Evict RAM_DISK block with open handle should fall back to DISK. > * Delete RAM_DISK block (persisted) with open handle should mark the block to > be deleted upon handle close. > Simply open handle to file in DFS name space won't work as expected. We need > a local FS file handle to the block file. The only meaningful case is for > Short Circuit Read. This JIRA is to validate/enable the two cases with SCR > enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[
https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144136#comment-14144136
]
Arpit Agarwal commented on HDFS-6990:
-
FTR, here is the offline feedback I had on this patch:
{code}
+ // Ensure path1 is still readable from the open SCR handle.
+ fis.read(fis.getPos(), buf, 0, BUFFER_LENGTH);
+ assertThat(verifyReadRandomFile(path1, BLOCK_SIZE, SEED), is(true));
{code}
This is not reading from the same handle as before. We should read from the fis
handle we opened earlier, and then also check the short circuit read counters
to ensure the read was short-circuited.
These tests should be skipped on Windows or when native IO is not enabled. You
could add something like:
{code}
assumeTrue(NativeCodeLoader.isNativeCodeLoaded() && !Path.WINDOWS);
{code}
See Chris's recent fix to HDFS-7110.
Also it is worth noting (maybe in a comment) we are not really testing SCR from
RAM_DISK, since the test fakes RAM_DISK on physical disk.
+1 with these updates.
> Add unit test for evict/delete RAM_DISK block with open handle
> --
>
> Key: HDFS-6990
> URL: https://issues.apache.org/jira/browse/HDFS-6990
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch,
> HDFS-6990.3.patch
>
>
> This is to verify:
> * Evict RAM_DISK block with open handle should fall back to DISK.
> * Delete RAM_DISK block (persisted) with open handle should mark the block to
> be deleted upon handle close.
> Simply open handle to file in DFS name space won't work as expected. We need
> a local FS file handle to the block file. The only meaningful case is for
> Short Circuit Read. This JIRA is to validate/enable the two cases with SCR
> enabled MiniDFSCluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6990) Add unit test for evict/delete RAM_DISK block with open handle
[ https://issues.apache.org/jira/browse/HDFS-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144132#comment-14144132 ] Arpit Agarwal commented on HDFS-6990: - +1 for the patch. Thanks for addressing all the rounds of feedback [~xyao]. I will commit it shortly. > Add unit test for evict/delete RAM_DISK block with open handle > -- > > Key: HDFS-6990 > URL: https://issues.apache.org/jira/browse/HDFS-6990 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao > Attachments: HDFS-6990.0.patch, HDFS-6990.1.patch, HDFS-6990.2.patch, > HDFS-6990.3.patch > > > This is to verify: > * Evict RAM_DISK block with open handle should fall back to DISK. > * Delete RAM_DISK block (persisted) with open handle should mark the block to > be deleted upon handle close. > Simply open handle to file in DFS name space won't work as expected. We need > a local FS file handle to the block file. The only meaningful case is for > Short Circuit Read. This JIRA is to validate/enable the two cases with SCR > enabled MiniDFSCluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
