[
https://issues.apache.org/jira/browse/HADOOP-18463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608873#comment-17608873
]
ASF GitHub Bot commented on HADOOP-18463:
-----------------------------------------
mukund-thakur commented on code in PR #4921:
URL: https://github.com/apache/hadoop/pull/4921#discussion_r978951742
##########
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractVectoredReadTest.java:
##########
@@ -364,6 +373,63 @@ public void testMultipleVectoredReads() throws Exception {
}
}
+ /**
+ * This test creates list of ranges and then submit a readVectored
+ * operation and then uses a separate thread pool to process the
+ * results asynchronously.
+ */
+ @Test
+ public void testVectoredIOEndToEnd() throws Exception {
+ FileSystem fs = getFileSystem();
+ List<FileRange> fileRanges = new ArrayList<>();
+ fileRanges.add(FileRange.createFileRange(8 * 1024, 100));
+ fileRanges.add(FileRange.createFileRange(14 * 1024, 100));
+ fileRanges.add(FileRange.createFileRange(10 * 1024, 100));
+ fileRanges.add(FileRange.createFileRange(2 * 1024 - 101, 100));
+ fileRanges.add(FileRange.createFileRange(40 * 1024, 1024));
+
+ ExecutorService dataProcessor = Executors.newFixedThreadPool(5);
+ CountDownLatch countDown = new CountDownLatch(fileRanges.size());
+
+ try (FSDataInputStream in = fs.open(path(VECTORED_READ_FILE_NAME))) {
+ in.readVectored(fileRanges, value -> pool.getBuffer(true, value));
+ // user can perform other computations while waiting for IO.
+ for (FileRange res : fileRanges) {
+ dataProcessor.submit(() -> {
+ try {
+ readBufferValidateDataAndReturnToPool(pool, res, countDown);
+ } catch (Exception e) {
+ LOG.error("Error while process result for {} ", res, e);
+ }
+ });
+ }
+ if (!countDown.await(100, TimeUnit.SECONDS)) {
+ throw new AssertionError("Error while processing vectored io results");
+ }
+ } finally {
+ pool.release();
Review Comment:
Actually I shouldn't be calling pool.release() here as it is a shared pool
by all the tests. Initially I started with a new pool for this specific test
then moved to the common one.
> Add an integration test to process data asynchronously during vectored read.
> ----------------------------------------------------------------------------
>
> Key: HADOOP-18463
> URL: https://issues.apache.org/jira/browse/HADOOP-18463
> Project: Hadoop Common
> Issue Type: Sub-task
> Reporter: Mukund Thakur
> Assignee: Mukund Thakur
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]