[ 
https://issues.apache.org/jira/browse/HADOOP-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608873#comment-17608873
 ] 

ASF GitHub Bot commented on HADOOP-18463:
-----------------------------------------

mukund-thakur commented on code in PR #4921:
URL: https://github.com/apache/hadoop/pull/4921#discussion_r978951742


##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java:
##########
@@ -364,6 +373,63 @@ public void testMultipleVectoredReads() throws Exception {
     }
   }
 
+  /**
+   * This test creates list of ranges and then submit a readVectored
+   * operation and then uses a separate thread pool to process the
+   * results asynchronously.
+   */
+  @Test
+  public void testVectoredIOEndToEnd() throws Exception {
+    FileSystem fs = getFileSystem();
+    List<FileRange> fileRanges = new ArrayList<>();
+    fileRanges.add(FileRange.createFileRange(8 * 1024, 100));
+    fileRanges.add(FileRange.createFileRange(14 * 1024, 100));
+    fileRanges.add(FileRange.createFileRange(10 * 1024, 100));
+    fileRanges.add(FileRange.createFileRange(2 * 1024 - 101, 100));
+    fileRanges.add(FileRange.createFileRange(40 * 1024, 1024));
+
+    ExecutorService dataProcessor = Executors.newFixedThreadPool(5);
+    CountDownLatch countDown = new CountDownLatch(fileRanges.size());
+
+    try (FSDataInputStream in = fs.open(path(VECTORED_READ_FILE_NAME))) {
+      in.readVectored(fileRanges, value -> pool.getBuffer(true, value));
+      // user can perform other computations while waiting for IO.
+      for (FileRange res : fileRanges) {
+        dataProcessor.submit(() -> {
+          try {
+            readBufferValidateDataAndReturnToPool(pool, res, countDown);
+          } catch (Exception e) {
+            LOG.error("Error while process result for {} ", res, e);
+          }
+        });
+      }
+      if (!countDown.await(100, TimeUnit.SECONDS)) {
+        throw new AssertionError("Error while processing vectored io results");
+      }
+    } finally {
+      pool.release();

Review Comment:
   Actually I shouldn't be calling pool.release() here as it is a shared pool 
by all the tests. Initially I started with a new pool for this specific test 
then moved to the common one. 





> Add an integration test to process data asynchronously during vectored read.
> ----------------------------------------------------------------------------
>
>                 Key: HADOOP-18463
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18463
>             Project: Hadoop Common
>          Issue Type: Sub-task
>            Reporter: Mukund Thakur
>            Assignee: Mukund Thakur
>            Priority: Major
>              Labels: pull-request-available
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to