Steve Loughran created HADOOP-18805:
---------------------------------------
Summary: s3a large file prefetch tests are too slow, don't
validate data
Key: HADOOP-18805
URL: https://issues.apache.org/jira/browse/HADOOP-18805
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.9
Reporter: Steve Loughran
the large file prefetch tests (including LRU cache eviction) are really slow.
moving under -scale may hide the problem for most runs, but they are still too
slow, can time out, etc etc.
also, and this is very, very important, they can't validate the data.
Better:
* test on smaller files by setting a very small block size (1k bytes or less)
just to force paged reads of a small 16k file.
* with known contents to the values of all forms of read can be validated
* maybe the LRU tests can work with a fake remote object which can then be used
in a unit test
* extend one of the huge file tests to read from there -including s3-CSE
encryption coverage.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
