Copilot commented on code in PR #11971:
URL: https://github.com/apache/ozone/pull/11971#discussion_r4566574829


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/OzoneFileSystemTestBase.java:
##########
@@ -167,6 +168,32 @@ static void listLocatedStatusForZeroByteFile(FileSystem 
fs, Path path) throws IO
     assertFalse(fileStatuses[0] instanceof LocatedFileStatus);
   }
 
+  /**
+   * Like HDFS, getFileStatus returns no block locations and 
getFileBlockLocations fetches them; for a file deleted
+   * after its getFileStatus, getFileBlockLocations fails with 
FileNotFoundException. Deletes the file.
+   */
+  static void fileBlockLocations(FileSystem fs, Path path, long length) throws 
IOException {
+    FileStatus status = fs.getFileStatus(path);
+    assertThat(status).isNotInstanceOf(LocatedFileStatus.class);
+    assertEquals(length, status.getLen());
+
+    BlockLocation[] expected = 
fs.listLocatedStatus(path).next().getBlockLocations();
+    assertThat(expected.length).isGreaterThanOrEqualTo(1);
+    for (BlockLocation[] blockLocations : new BlockLocation[][] {
+        fs.getFileBlockLocations(status, 0, status.getLen()), 
fs.getFileBlockLocations(path, 0, status.getLen())}) {
+      assertEquals(expected.length, blockLocations.length);
+      for (int i = 0; i < expected.length; i++) {
+        
assertThat(blockLocations[i].getNames().length).isGreaterThanOrEqualTo(1);
+        assertArrayEquals(expected[i].getHosts(), 
blockLocations[i].getHosts());

Review Comment:
   Asserting `assertArrayEquals` on `getHosts()` can be order-sensitive and may 
become flaky if host ordering differs between the two call paths being 
compared. To make the integration test more robust, compare hosts as unordered 
sets (or sort both arrays before comparing) while keeping the offset/length 
assertions.



##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -887,9 +892,30 @@ public BlockLocation[] getFileBlockLocations(FileStatus 
fileStatus,
       throws IOException {
     if (fileStatus instanceof LocatedFileStatus) {
       return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start, 
len);
-    } else {
-      return super.getFileBlockLocations(fileStatus, start, len);
     }
+    if (needsBlockLocations(fileStatus, start, len)) {
+      // getFileStatus returns no block locations: fetch them by path, as HDFS 
does.
+      return getFileBlockLocations(fileStatus.getPath(), start, len);
+    }
+    return super.getFileBlockLocations(fileStatus, start, len);
+  }
+
+  @Override
+  public BlockLocation[] getFileBlockLocations(Path p, long start, long len) 
throws IOException {
+    // One GetFileStatus with block locations, instead of getFileStatus 
followed by another for the locations.
+    FileStatus fileStatus = getFileStatus(p, false);
+    if (fileStatus instanceof LocatedFileStatus) {
+      return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start, 
len);
+    }
+    return super.getFileBlockLocations(fileStatus, start, len);
+  }

Review Comment:
   This path-based overload performs an OM call (`getFileStatus(p, false)`) 
before validating `start/len`. For invalid inputs (eg. `start < 0` or `len < 
0`), you can throw `IllegalArgumentException` immediately to avoid an 
unnecessary RPC and keep behavior consistent with the `FileStatus` overload 
tests.



##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -887,9 +892,30 @@ public BlockLocation[] getFileBlockLocations(FileStatus 
fileStatus,
       throws IOException {
     if (fileStatus instanceof LocatedFileStatus) {
       return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start, 
len);
-    } else {
-      return super.getFileBlockLocations(fileStatus, start, len);
     }
+    if (needsBlockLocations(fileStatus, start, len)) {
+      // getFileStatus returns no block locations: fetch them by path, as HDFS 
does.
+      return getFileBlockLocations(fileStatus.getPath(), start, len);
+    }
+    return super.getFileBlockLocations(fileStatus, start, len);
+  }
+
+  @Override
+  public BlockLocation[] getFileBlockLocations(Path p, long start, long len) 
throws IOException {
+    // One GetFileStatus with block locations, instead of getFileStatus 
followed by another for the locations.
+    FileStatus fileStatus = getFileStatus(p, false);
+    if (fileStatus instanceof LocatedFileStatus) {
+      return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start, 
len);
+    }
+    return super.getFileBlockLocations(fileStatus, start, len);
+  }
+
+  /**
+   * Whether the block locations of the range must be fetched for a status 
from {@link #getFileStatus(Path)}.
+   */
+  static boolean needsBlockLocations(FileStatus fileStatus, long start, long 
len) {
+    return fileStatus instanceof FileStatusWithoutBlockLocations && 
fileStatus.isFile()
+        && start >= 0 && len >= 0 && start < fileStatus.getLen();
   }

Review Comment:
   `needsBlockLocations` currently returns true when `len == 0`, which will 
trigger an OM call even though a zero-length range should return an empty 
result. Consider adding `len > 0` (and possibly also guarding against 
`fileStatus.getLen() == 0`) so `getFileBlockLocations(status, start, 0)` 
short-circuits without fetching block locations.



##########
hadoop-ozone/ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemHeadOp.java:
##########
@@ -48,6 +54,12 @@
  */
 public class TestOzoneFileSystemHeadOp {
 
+  private static final BlockLocation[] BLOCKS = {
+      new BlockLocation(new String[] {"dn1:9859"}, new String[] {"dn1"}, 0, 
100),
+      new BlockLocation(new String[] {"dn2:9859"}, new String[] {"dn2"}, 100, 
100),
+      new BlockLocation(new String[] {"dn3:9859"}, new String[] {"dn3"}, 200, 
50),
+  };

Review Comment:
   This `BLOCKS` fixture (and the related stubbing/verification helpers) 
appears duplicated in `TestRootedOzoneFileSystemHeadOp`. Consider extracting 
the shared fixtures/helpers into a small test utility or a shared base to 
reduce duplication and keep future behavior changes in one place.



##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/FileStatusWithoutBlockLocations.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.ozone;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * The status that ofs and o3fs {@code getFileStatus} return. Like HDFS, it 
has no block locations, and
+ * {@code getFileBlockLocations} fetches them from OM for it.
+ * <p>
+ * A status from {@code listStatus} has no block locations either, but {@code 
getFileBlockLocations} keeps returning a
+ * placeholder location for it without an OM call, so that a listing that asks 
for the locations of every file (as
+ * Spark does) does not pay an OM call per file.
+ */
+final class FileStatusWithoutBlockLocations extends FileStatus {
+  private static final long serialVersionUID = 1L;
+
+  private final boolean encrypted;
+  private final boolean erasureCoded;
+
+  /**
+   * @param erasureCoded {@code status.isErasureCoded()}, which is not in 
Hadoop 2
+   */
+  FileStatusWithoutBlockLocations(FileStatus status, boolean erasureCoded) 
throws IOException {
+    // the copy constructor does not copy the encrypted and erasure coded flags
+    super(status);
+    this.encrypted = status.isEncrypted();
+    this.erasureCoded = erasureCoded;
+  }
+
+  @Override
+  public boolean isEncrypted() {
+    return encrypted;
+  }
+
+  @Override
+  public boolean isErasureCoded() {
+    return erasureCoded;
+  }

Review Comment:
   The constructor Javadoc says `status.isErasureCoded()` is 'not in Hadoop 2', 
but this class both calls `status.isEncrypted()` and declares `@Override 
isErasureCoded()`. If Hadoop 2 compatibility is a requirement here, the comment 
should be clarified to the specific Hadoop 2 versions (or removed if outdated) 
so it doesn’t contradict the implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to