This is an automated email from the ASF dual-hosted git repository.
mattcasters pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new a57df2fb1a Issue #9113 : List local Spark directories without Hadoop's
native Windows library (#9114)
a57df2fb1a is described below
commit a57df2fb1aae22505db27364ebce902ff92c4897
Author: Matt Casters <[email protected]>
AuthorDate: Thu Oct 29 22:45:10 2026 +0100
Issue #9113 : List local Spark directories without Hadoop's native Windows
library (#9114)
---
.../pipeline/spark/paths-and-filesystems.adoc | 4 +-
.../hop/spark/fs/HopWindowsLocalFileSystem.java | 95 ++++++++++++++++++++--
.../spark/fs/HopWindowsLocalFileSystemTest.java | 69 ++++++++++++++++
3 files changed, 159 insertions(+), 9 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
index d5107214d3..6152c9eac8 100644
---
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
+++
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/spark/paths-and-filesystems.adoc
@@ -83,7 +83,9 @@ For the full list of Hop VFS schemes, see
xref:vfs.adoc[Virtual File Systems].
On Windows, Hadoop's local file system sets POSIX permissions by running
`winutils.exe` from `HADOOP_HOME` (or the `hadoop.home.dir` system property).
Spark file output hits that call when the commit creates the output directory.
With neither variable set, the write fails with `HADOOP_HOME and
hadoop.home.dir are unset`.
-When Hop creates the Spark session on Windows and `winutils.exe` is not
installed, local `file:///` reads and writes skip those permission changes. The
files keep the permissions of the Hop process. You do not need a Hadoop
installation for `local[*]`.
+When Hop creates the Spark session on Windows and `winutils.exe` is not
installed, local `file:///` reads and writes skip those permission changes.
Listing a directory during the output commit uses Java file checks as well.
Hadoop's own listing calls `NativeIO.Windows.access`, which needs `hadoop.dll`
and throws `UnsatisfiedLinkError` when that library is not installed. The files
keep the permissions of the Hop process. You do not need a Hadoop installation
for `local[*]`.
+
+A `file:///` path is this same local file system. A variable such as
`+${PROJECT_HOME}+` is often stored as a `file:///` URI, and resolving it keeps
that prefix. Spark File Output passes the resolved URI to Hadoop. A drive path
such as `C:/output` uses the same file system.
If `HADOOP_HOME` points at a directory that contains `bin/winutils.exe`, Hop
keeps Hadoop's own local file system and permissions are applied. To choose a
file system yourself, set `spark.hadoop.fs.file.impl` in the run configuration.
That entry wins.
diff --git
a/plugins/engines/spark/src/main/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystem.java
b/plugins/engines/spark/src/main/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystem.java
index 0cf0a2c319..843b327d14 100644
---
a/plugins/engines/spark/src/main/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystem.java
+++
b/plugins/engines/spark/src/main/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystem.java
@@ -17,7 +17,11 @@
package org.apache.hop.spark.fs;
+import java.io.File;
+import java.io.FileNotFoundException;
import java.io.IOException;
+import java.nio.file.AccessDeniedException;
+import java.util.Arrays;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.LocalFileSystem;
import org.apache.hadoop.fs.Path;
@@ -37,10 +41,18 @@ import org.apache.spark.sql.SparkSession;
* hadoop.home.dir}, and the call fails with {@code HADOOP_HOME and
hadoop.home.dir are unset} when
* neither is set. The file itself was already created by Java; only the
permission change fails.
*
- * <p>This class keeps the checksummed local file system and skips those
permission changes. Files
- * keep the permissions of the Hop process. It is installed only when Hop
creates the session, and
- * only when Windows has neither winutils nor Hadoop's native I/O library. An
explicit {@code
- * spark.hadoop.fs.file.impl} in the run configuration is applied afterwards
and wins.
+ * <p>This class keeps the checksummed local file system and skips those
permission changes. It also
+ * lists directories with {@link File} instead of {@code FileUtil.list}. On
Windows that helper
+ * calls {@code NativeIO.Windows.access} without checking that the native
library loaded, and the
+ * resulting {@link UnsatisfiedLinkError} is not an {@link IOException}. Spark
file output hits it
+ * when {@code FileOutputCommitter} lists the output directory. A {@code
file:///} path, including
+ * one produced by resolving a VFS variable such as {@code PROJECT_HOME}, is
this same local file
+ * system.
+ *
+ * <p>Files keep the permissions of the Hop process. The class is installed
only when Hop creates
+ * the session, and only when Windows has neither winutils nor Hadoop's native
I/O library. An
+ * explicit {@code spark.hadoop.fs.file.impl} in the run configuration is
applied afterwards and
+ * wins. A reused session keeps the file system it was created with.
*/
public class HopWindowsLocalFileSystem extends LocalFileSystem {
@@ -48,7 +60,12 @@ public class HopWindowsLocalFileSystem extends
LocalFileSystem {
public static final String SPARK_HADOOP_FILE_IMPL =
"spark.hadoop.fs.file.impl";
public HopWindowsLocalFileSystem() {
- super(new WinutilsFreeRawLocalFileSystem());
+ this(new WinutilsFreeRawLocalFileSystem());
+ }
+
+ /** Visible to tests that supply a raw file system with the Java directory
listing forced on. */
+ HopWindowsLocalFileSystem(WinutilsFreeRawLocalFileSystem raw) {
+ super(raw);
}
/**
@@ -89,18 +106,80 @@ public class HopWindowsLocalFileSystem extends
LocalFileSystem {
if (log != null) {
log.logBasic(
"HADOOP_HOME and hadoop.home.dir are unset. Local Spark paths will
keep the"
- + " permissions of this process instead of calling winutils.");
+ + " permissions of this process instead of calling winutils or
Hadoop's native"
+ + " Windows library.");
}
}
/**
- * Raw local file system that does not shell out for chmod or for owner
lookups.
+ * Raw local file system that does not shell out for chmod or for owner
lookups, and does not call
+ * the Windows native access check while listing a directory.
*
* <p>{@code mkdirs} and {@code create} call {@code setPermission} on this
instance, not on the
- * checksum wrapper, so the override has to live here.
+ * checksum wrapper. {@code ChecksumFileSystem.listStatus} calls {@code
listStatus} here. Both
+ * overrides have to live on this class.
*/
public static class WinutilsFreeRawLocalFileSystem extends
RawLocalFileSystem {
+ @Override
+ public FileStatus[] listStatus(Path f) throws IOException {
+ if (!skipNativeDirectoryListing()) {
+ return super.listStatus(f);
+ }
+ File localFile = pathToFile(f);
+ if (!localFile.exists()) {
+ throw new FileNotFoundException("File " + f + " does not exist");
+ }
+ if (!localFile.isDirectory()) {
+ return new FileStatus[] {getFileStatus(f)};
+ }
+ String[] names = listLocalNames(localFile);
+ FileStatus[] results = new FileStatus[names.length];
+ int kept = 0;
+ for (String name : names) {
+ try {
+ // The three-argument Path keeps a colon in a name from being read
as a scheme.
+ results[kept] = getFileStatus(new Path(f, new Path(null, null,
name)));
+ kept++;
+ } catch (FileNotFoundException ignored) {
+ // The directory changed between list() and getFileStatus().
+ }
+ }
+ if (kept == names.length) {
+ return results;
+ }
+ return Arrays.copyOf(results, kept);
+ }
+
+ /**
+ * Whether directory listing should use {@link File} instead of {@code
FileUtil.list}.
+ *
+ * <p>The default follows {@link HopWindowsLocalFileSystem#needed()}.
Tests override it so the
+ * Java listing can run on Linux, where {@code needed()} is false and
{@code FileUtil.canRead}
+ * does not call the Windows native method.
+ */
+ boolean skipNativeDirectoryListing() {
+ return HopWindowsLocalFileSystem.needed();
+ }
+
+ /**
+ * Names in a directory, using {@link File#canRead()} and {@link
File#list()}.
+ *
+ * <p>{@code FileUtil.list} refuses the directory through {@code
FileUtil.canRead}. On Windows
+ * that method always calls {@code NativeIO.Windows.access}. When {@code
hadoop.dll} is not
+ * loaded the native method throws {@link UnsatisfiedLinkError}.
+ */
+ static String[] listLocalNames(File directory) throws IOException {
+ if (!directory.canRead()) {
+ throw new AccessDeniedException(directory.toString(), null,
"Permission denied");
+ }
+ String[] names = directory.list();
+ if (names == null) {
+ throw new IOException("Invalid directory or I/O error occurred for
dir: " + directory);
+ }
+ return names;
+ }
+
@Override
public void setPermission(Path path, FsPermission permission) throws
IOException {
if (HopWindowsLocalFileSystem.needed()) {
diff --git
a/plugins/engines/spark/src/test/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystemTest.java
b/plugins/engines/spark/src/test/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystemTest.java
index 43d51a58ca..7ec5856dfc 100644
---
a/plugins/engines/spark/src/test/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystemTest.java
+++
b/plugins/engines/spark/src/test/java/org/apache/hop/spark/fs/HopWindowsLocalFileSystemTest.java
@@ -20,10 +20,16 @@ package org.apache.hop.spark.fs;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
+import java.io.File;
import java.net.URI;
import java.nio.charset.StandardCharsets;
+import java.nio.file.AccessDeniedException;
+import java.util.Arrays;
+import java.util.Set;
+import java.util.stream.Collectors;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
@@ -31,6 +37,7 @@ import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.permission.FsPermission;
import org.apache.hadoop.util.Shell;
+import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
@@ -78,6 +85,68 @@ class HopWindowsLocalFileSystemTest {
}
}
+ @Test
+ void listsAFileUriWithJavaInsteadOfTheNativeWindowsAccessCheck() throws
Exception {
+ java.nio.file.Path localDir = tempDir.resolve("commit-out");
+ java.nio.file.Files.createDirectories(localDir);
+ java.nio.file.Files.writeString(localDir.resolve("part-00000.csv"), "a,b");
+ java.nio.file.Files.writeString(localDir.resolve(".part-00000.csv.crc"),
"crc");
+
+ // The three-slash form is what a resolved VFS variable looks like
(file:///C:/... on Windows).
+ String fileUri = localDir.toUri().toString();
+ assertTrue(fileUri.startsWith("file:///"), fileUri);
+ Path dir = new Path(fileUri);
+
+ HopWindowsLocalFileSystem fs = new HopWindowsLocalFileSystem(new
JavaDirectoryListing());
+ try {
+ fs.initialize(URI.create("file:///"), new Configuration());
+ FileStatus[] listed = fs.listStatus(dir);
+ Set<String> names =
+ Arrays.stream(listed)
+ .map(status -> status.getPath().getName())
+ .collect(Collectors.toSet());
+ assertEquals(Set.of("part-00000.csv"), names);
+
+ FileStatus[] oneFile = fs.listStatus(new Path(dir, "part-00000.csv"));
+ assertEquals(1, oneFile.length);
+ assertEquals("part-00000.csv", oneFile[0].getPath().getName());
+
+ assertThrows(
+ java.io.FileNotFoundException.class, () -> fs.listStatus(new
Path(dir, "missing")));
+ } finally {
+ fs.close();
+ }
+ }
+
+ @Test
+ void javaDirectoryListingRefusesAnUnreadableDirectory() throws Exception {
+ File directory = tempDir.resolve("hidden").toFile();
+ assertTrue(directory.mkdir());
+ directory.setReadable(false);
+ try {
+ Assumptions.assumeFalse(directory.canRead(), "this user can still read
the directory");
+ AccessDeniedException denied =
+ assertThrows(
+ AccessDeniedException.class,
+ () ->
+
HopWindowsLocalFileSystem.WinutilsFreeRawLocalFileSystem.listLocalNames(
+ directory));
+ assertTrue(denied.getMessage().contains(directory.toString()));
+ } finally {
+ directory.setReadable(true);
+ }
+ }
+
+ /** Raw file system that always takes the Windows listing path, including on
Linux. */
+ private static final class JavaDirectoryListing
+ extends HopWindowsLocalFileSystem.WinutilsFreeRawLocalFileSystem {
+
+ @Override
+ boolean skipNativeDirectoryListing() {
+ return true;
+ }
+ }
+
private static String read(HopWindowsLocalFileSystem fs, Path file) throws
Exception {
byte[] buf = new byte[5];
try (FSDataInputStream in = fs.open(file)) {
