mattcasters opened a new pull request, #8951: URL: https://github.com/apache/hop/pull/8951
## Summary Spark file output on Windows `local[*]` fails while Hadoop creates the output directory. `RawLocalFileSystem.setPermission` runs `winutils.exe`, and that lookup throws `HADOOP_HOME and hadoop.home.dir are unset` when neither is set. The same path is still in main. When Hop creates the Spark session on Windows and winutils is not installed, the session uses a local file system that skips those permission changes. Files keep the permissions of the Hop process. `hdfs://` and `s3a://` are unchanged. If `HADOOP_HOME` already points at `bin/winutils.exe`, Hadoop's own local file system stays in place. A `spark.hadoop.fs.file.impl` line in the run configuration is applied after this default and wins. The help-button 404 does not apply to main. Spark file input and Spark file output were undocumented in 2.19, so `/manual/latest/` 404s. The pages are in main and published at `/manual/next/`. A 2.20 snapshot build opens that version. Fixes #8946 ## Test plan - [x] `HopWindowsLocalFileSystemTest` (2) on Linux: decision table, and mkdir/create/setPermission/read through the wrapper - [x] Checkstyle and Apache RAT on `hop-engines-spark` - [ ] Windows `local[*]` Spark file output to a `file:///` directory with `HADOOP_HOME` unset -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
