hansva opened a new issue, #9591: URL: https://github.com/apache/hop/issues/9591
### Apache Hop version? 2.20.0 ### Java version? 21 ### Operating system Other ### What happened? A pipeline run with the native Spark engine doesn't find the named VFS connections defined in the project, such as MinIO, S3 or Azure connections. A path like `minio-connection://bucket/file` doesn't give a "connection not found" error. It quietly resolves to a local folder named `minio-connection:`. - An input transform then reads nothing, or fails on a file that doesn't exist. - An output transform writes into the local folder, and nothing ever reaches the bucket. **Why** The local pipeline engine opens a VFS namespace for the metadata of its run, so the project's named connections resolve while the pipeline runs. The native Spark engine (`SparkPipelineEngine`) never opens one. It falls back to the file system manager for the whole Hop process, which doesn't always know the project's connections. **How to reproduce** 1. In a project, create a named VFS connection, for example a MinIO connection called `minio-connection`. 2. Build a pipeline that writes a text file to `minio-connection://bucket/out.csv`. 3. Run it with the local engine: the file ends up in the bucket. 4. Run it with a native Spark run configuration (`local[*]` is enough): no file in the bucket, and a local `minio-connection:` folder appears. **Fix** `SparkPipelineEngine` now does the same as the local engine: - It opens the VFS namespace in `prepareExecution()`. - It binds the namespace to the threads that run the pipeline. - It closes the namespace when the run finishes, when `cleanup()` is called, or when preparing the run fails. Closing is guarded, so the namespace is never closed twice. A run that uses the process metadata gets no namespace of its own, and nothing changes for it. ### Issue Priority Priority: 2 ### Issue Component Component: Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
