Copilot commented on code in PR #11971:
URL: https://github.com/apache/ozone/pull/11971#discussion_r4566574829
##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/OzoneFileSystemTestBase.java:
##########
@@ -167,6 +168,32 @@ static void listLocatedStatusForZeroByteFile(FileSystem
fs, Path path) throws IO
assertFalse(fileStatuses[0] instanceof LocatedFileStatus);
}
+ /**
+ * Like HDFS, getFileStatus returns no block locations and
getFileBlockLocations fetches them; for a file deleted
+ * after its getFileStatus, getFileBlockLocations fails with
FileNotFoundException. Deletes the file.
+ */
+ static void fileBlockLocations(FileSystem fs, Path path, long length) throws
IOException {
+ FileStatus status = fs.getFileStatus(path);
+ assertThat(status).isNotInstanceOf(LocatedFileStatus.class);
+ assertEquals(length, status.getLen());
+
+ BlockLocation[] expected =
fs.listLocatedStatus(path).next().getBlockLocations();
+ assertThat(expected.length).isGreaterThanOrEqualTo(1);
+ for (BlockLocation[] blockLocations : new BlockLocation[][] {
+ fs.getFileBlockLocations(status, 0, status.getLen()),
fs.getFileBlockLocations(path, 0, status.getLen())}) {
+ assertEquals(expected.length, blockLocations.length);
+ for (int i = 0; i < expected.length; i++) {
+
assertThat(blockLocations[i].getNames().length).isGreaterThanOrEqualTo(1);
+ assertArrayEquals(expected[i].getHosts(),
blockLocations[i].getHosts());
Review Comment:
Asserting `assertArrayEquals` on `getHosts()` can be order-sensitive and may
become flaky if host ordering differs between the two call paths being
compared. To make the integration test more robust, compare hosts as unordered
sets (or sort both arrays before comparing) while keeping the offset/length
assertions.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -887,9 +892,30 @@ public BlockLocation[] getFileBlockLocations(FileStatus
fileStatus,
throws IOException {
if (fileStatus instanceof LocatedFileStatus) {
return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start,
len);
- } else {
- return super.getFileBlockLocations(fileStatus, start, len);
}
+ if (needsBlockLocations(fileStatus, start, len)) {
+ // getFileStatus returns no block locations: fetch them by path, as HDFS
does.
+ return getFileBlockLocations(fileStatus.getPath(), start, len);
+ }
+ return super.getFileBlockLocations(fileStatus, start, len);
+ }
+
+ @Override
+ public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
throws IOException {
+ // One GetFileStatus with block locations, instead of getFileStatus
followed by another for the locations.
+ FileStatus fileStatus = getFileStatus(p, false);
+ if (fileStatus instanceof LocatedFileStatus) {
+ return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start,
len);
+ }
+ return super.getFileBlockLocations(fileStatus, start, len);
+ }
Review Comment:
This path-based overload performs an OM call (`getFileStatus(p, false)`)
before validating `start/len`. For invalid inputs (eg. `start < 0` or `len <
0`), you can throw `IllegalArgumentException` immediately to avoid an
unnecessary RPC and keep behavior consistent with the `FileStatus` overload
tests.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -887,9 +892,30 @@ public BlockLocation[] getFileBlockLocations(FileStatus
fileStatus,
throws IOException {
if (fileStatus instanceof LocatedFileStatus) {
return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start,
len);
- } else {
- return super.getFileBlockLocations(fileStatus, start, len);
}
+ if (needsBlockLocations(fileStatus, start, len)) {
+ // getFileStatus returns no block locations: fetch them by path, as HDFS
does.
+ return getFileBlockLocations(fileStatus.getPath(), start, len);
+ }
+ return super.getFileBlockLocations(fileStatus, start, len);
+ }
+
+ @Override
+ public BlockLocation[] getFileBlockLocations(Path p, long start, long len)
throws IOException {
+ // One GetFileStatus with block locations, instead of getFileStatus
followed by another for the locations.
+ FileStatus fileStatus = getFileStatus(p, false);
+ if (fileStatus instanceof LocatedFileStatus) {
+ return getBlockLocationsInRange((LocatedFileStatus) fileStatus, start,
len);
+ }
+ return super.getFileBlockLocations(fileStatus, start, len);
+ }
+
+ /**
+ * Whether the block locations of the range must be fetched for a status
from {@link #getFileStatus(Path)}.
+ */
+ static boolean needsBlockLocations(FileStatus fileStatus, long start, long
len) {
+ return fileStatus instanceof FileStatusWithoutBlockLocations &&
fileStatus.isFile()
+ && start >= 0 && len >= 0 && start < fileStatus.getLen();
}
Review Comment:
`needsBlockLocations` currently returns true when `len == 0`, which will
trigger an OM call even though a zero-length range should return an empty
result. Consider adding `len > 0` (and possibly also guarding against
`fileStatus.getLen() == 0`) so `getFileBlockLocations(status, start, 0)`
short-circuits without fetching block locations.
##########
hadoop-ozone/ozonefs-common/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemHeadOp.java:
##########
@@ -48,6 +54,12 @@
*/
public class TestOzoneFileSystemHeadOp {
+ private static final BlockLocation[] BLOCKS = {
+ new BlockLocation(new String[] {"dn1:9859"}, new String[] {"dn1"}, 0,
100),
+ new BlockLocation(new String[] {"dn2:9859"}, new String[] {"dn2"}, 100,
100),
+ new BlockLocation(new String[] {"dn3:9859"}, new String[] {"dn3"}, 200,
50),
+ };
Review Comment:
This `BLOCKS` fixture (and the related stubbing/verification helpers)
appears duplicated in `TestRootedOzoneFileSystemHeadOp`. Consider extracting
the shared fixtures/helpers into a small test utility or a shared base to
reduce duplication and keep future behavior changes in one place.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/FileStatusWithoutBlockLocations.java:
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.ozone;
+
+import java.io.IOException;
+import org.apache.hadoop.fs.FileStatus;
+
+/**
+ * The status that ofs and o3fs {@code getFileStatus} return. Like HDFS, it
has no block locations, and
+ * {@code getFileBlockLocations} fetches them from OM for it.
+ * <p>
+ * A status from {@code listStatus} has no block locations either, but {@code
getFileBlockLocations} keeps returning a
+ * placeholder location for it without an OM call, so that a listing that asks
for the locations of every file (as
+ * Spark does) does not pay an OM call per file.
+ */
+final class FileStatusWithoutBlockLocations extends FileStatus {
+ private static final long serialVersionUID = 1L;
+
+ private final boolean encrypted;
+ private final boolean erasureCoded;
+
+ /**
+ * @param erasureCoded {@code status.isErasureCoded()}, which is not in
Hadoop 2
+ */
+ FileStatusWithoutBlockLocations(FileStatus status, boolean erasureCoded)
throws IOException {
+ // the copy constructor does not copy the encrypted and erasure coded flags
+ super(status);
+ this.encrypted = status.isEncrypted();
+ this.erasureCoded = erasureCoded;
+ }
+
+ @Override
+ public boolean isEncrypted() {
+ return encrypted;
+ }
+
+ @Override
+ public boolean isErasureCoded() {
+ return erasureCoded;
+ }
Review Comment:
The constructor Javadoc says `status.isErasureCoded()` is 'not in Hadoop 2',
but this class both calls `status.isEncrypted()` and declares `@Override
isErasureCoded()`. If Hadoop 2 compatibility is a requirement here, the comment
should be clarified to the specific Hadoop 2 versions (or removed if outdated)
so it doesn’t contradict the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]