smengcl commented on code in PR #11971:
URL: https://github.com/apache/ozone/pull/11971#discussion_r4560612606
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractRootedOzoneFileSystemTest.java:
##########
@@ -280,6 +282,35 @@ void testUserHomeDirectory() {
userOfs.getHomeDirectory());
}
+ /**
+ * Like HDFS, getFileStatus has no block locations (HDDS-17714):
getFileBlockLocations fetches them, for a status
+ * and for a path.
+ */
+ @Test
+ void testGetFileBlockLocations() throws Exception {
+ Path path = new Path(bucketPath, "testGetFileBlockLocations");
+ byte[] data = ContractTestUtils.dataset(1024, 'a', 26);
+ ContractTestUtils.createFile(fs, path, true, data);
+
+ FileStatus status = fs.getFileStatus(path);
+ assertFalse(status instanceof LocatedFileStatus);
+ assertEquals(data.length, status.getLen());
+
+ BlockLocation[] expected =
fs.listLocatedStatus(path).next().getBlockLocations();
+ assertThat(expected.length).isGreaterThanOrEqualTo(1);
+ for (BlockLocation[] blockLocations : new BlockLocation[][] {
+ fs.getFileBlockLocations(status, 0, status.getLen()),
fs.getFileBlockLocations(path, 0, status.getLen())}) {
+ assertEquals(expected.length, blockLocations.length);
+ for (int i = 0; i < expected.length; i++) {
+
assertThat(blockLocations[i].getNames().length).isGreaterThanOrEqualTo(1);
+ assertArrayEquals(expected[i].getHosts(),
blockLocations[i].getHosts());
+ assertEquals(expected[i].getOffset(), blockLocations[i].getOffset());
+ assertEquals(expected[i].getLength(), blockLocations[i].getLength());
+ }
+ }
+ fs.delete(path, false);
Review Comment:
Since `getFileBlockLocations(status)` now goes back to OM by path, a status
taken before a delete gets a `FileNotFoundException` instead of the old cached
locations (same as HDFS). Could we pin that here with something like
`assertThrows(FileNotFoundException.class, () ->
fs.getFileBlockLocations(status, 0, status.getLen()))` right after the delete?
It reuses the shared cluster and covers the negative path.
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/AbstractOzoneFileSystemTest.java:
##########
@@ -1859,20 +1860,29 @@ void testOzoneManagerFileSystemInterface() throws
IOException {
}
@Test
- public void testOzoneManagerLocatedFileStatus() throws IOException {
+ public void testOzoneManagerFileBlockLocations() throws IOException {
String data = RandomStringUtils.secure().nextAlphanumeric(20);
String filePath = RandomStringUtils.secure().nextAlphanumeric(5);
Path path = createPath("/" + filePath);
try (FSDataOutputStream stream = fs.create(path)) {
stream.writeBytes(data);
}
+ // like HDFS, getFileStatus has no block locations, getFileBlockLocations
fetches them
FileStatus status = fs.getFileStatus(path);
- LocatedFileStatus locatedFileStatus =
assertInstanceOf(LocatedFileStatus.class, status);
-
assertThat(locatedFileStatus.getBlockLocations().length).isGreaterThanOrEqualTo(1);
-
- for (BlockLocation blockLocation : locatedFileStatus.getBlockLocations()) {
- assertThat(blockLocation.getNames().length).isGreaterThanOrEqualTo(1);
- assertThat(blockLocation.getHosts().length).isGreaterThanOrEqualTo(1);
+ assertFalse(status instanceof LocatedFileStatus);
Review Comment:
nit: the checks from here to the end of the loop are the same as in
`AbstractRootedOzoneFileSystemTest#testGetFileBlockLocations`. Both classes
extend `OzoneFileSystemTestBase`, so maybe move them into a static helper
there, next to `listLocatedStatusForZeroByteFile`. While at it,
`assertThat(status).isNotInstanceOf(LocatedFileStatus.class)` would give a
clearer failure message than `assertFalse(status instanceof LocatedFileStatus)`.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java:
##########
@@ -1080,21 +1080,28 @@ public long getDefaultBlockSize() {
OZONE_SCM_BLOCK_SIZE, OZONE_SCM_BLOCK_SIZE_DEFAULT, StorageUnit.BYTES);
}
+ /**
+ * Like HDFS, the status has no block locations: OM skips the pipeline
refresh (headOp).
+ * {@link #getFileBlockLocations} fetches the locations when they are asked
for.
+ */
@Override
public FileStatus getFileStatus(Path f) throws IOException {
Review Comment:
Code that casts the result of `getFileStatus` to `LocatedFileStatus` will
break after this change, so it may be worth adding a short release note on
HDDS-17714 to help downstream users notice it on upgrade.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]