fuatbasik commented on code in PR #7763:
URL: https://github.com/apache/hadoop/pull/7763#discussion_r2182645332
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAnalyticsAcceleratorStreamReading.java:
##########
@@ -194,4 +223,96 @@ public void testInvalidConfigurationThrows() throws
Exception {
() ->
S3SeekableInputStreamConfiguration.fromConfiguration(connectorConfiguration));
}
+ /**
+ *
+ * TXT files are classified as SEQUENTIAL format and use
SequentialPrefetcher(requests the entire 10MB file)
+ * RangeOptimiser splits ranges larger than maxRangeSizeBytes (8MB) using
partSizeBytes (8MB)
+ * The 10MB range gets split into: [0-8MB) and [8MB-10MB)
+ * Each split range becomes a separate Block, resulting in 2 GET requests:
+ */
+ @Test
+ public void testLargeFileMultipleGets() throws Throwable {
+ describe("Large file should trigger multiple GET requests");
+
+ Path dest = path("large-test-file.txt");
+ byte[] data = dataset(10 * S_1M, 256, 255);
+ writeDataset(getFileSystem(), dest, data, 10 * S_1M, 1024, true);
+
+ byte[] buffer = new byte[S_1M * 10];
+ try (FSDataInputStream inputStream = getFileSystem().open(dest)) {
+ IOStatistics ioStats = inputStream.getIOStatistics();
+ inputStream.readFully(buffer);
+
+ verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_GET_REQUESTS,
2);
Review Comment:
Well, the number of requests here would highly depend on how AAL implements
request splitting. If we change the default from 8MB to 16Mb tomorrow, these
tests will start failing. I guess this is OK for now but we might want to
explicitly set AAL request size configuration to ensure underlying changes does
not cause test failures here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
