hansva opened a new issue, #9591:
URL: https://github.com/apache/hop/issues/9591

   ### Apache Hop version?
   
   2.20.0
   
   ### Java version?
   
   21
   
   ### Operating system
   
   Other
   
   ### What happened?
   
   A pipeline run with the native Spark engine doesn't find the named VFS 
connections defined in the project, such as MinIO, S3 or Azure connections. A 
path like `minio-connection://bucket/file` doesn't give a "connection not 
found" error. It quietly resolves to a local folder named `minio-connection:`.
   - An input transform then reads nothing, or fails on a file that doesn't 
exist.
   - An output transform writes into the local folder, and nothing ever reaches 
the bucket.
   
   **Why**
   
   The local pipeline engine opens a VFS namespace for the metadata of its run, 
so the project's named connections resolve while the pipeline runs. The native 
Spark engine (`SparkPipelineEngine`) never opens one. It falls back to the file 
system manager for the whole Hop process, which doesn't always know the 
project's connections.
   
   **How to reproduce**
   
   1. In a project, create a named VFS connection, for example a MinIO 
connection called `minio-connection`.
   2. Build a pipeline that writes a text file to 
`minio-connection://bucket/out.csv`.
   3. Run it with the local engine: the file ends up in the bucket.
   4. Run it with a native Spark run configuration (`local[*]` is enough): no 
file in the bucket, and a local `minio-connection:` folder appears.
   
   **Fix**
   
   `SparkPipelineEngine` now does the same as the local engine:
   - It opens the VFS namespace in `prepareExecution()`.
   - It binds the namespace to the threads that run the pipeline.
   - It closes the namespace when the run finishes, when `cleanup()` is called, 
or when preparing the run fails.
   
   Closing is guarded, so the namespace is never closed twice. A run that uses 
the process metadata gets no namespace of its own, and nothing changes for it.
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Spark


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to