bamaer commented on code in PR #9592:
URL: https://github.com/apache/hop/pull/9592#discussion_r4643436745
##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/engines/SparkPipelineEngine.java:
##########
@@ -291,11 +305,13 @@ public void prepareExecution() throws HopException {
@Override
public void startThreads() throws HopException {
+ HopVfsNamespace previousNamespace = null;
ClassLoader pluginCl = getClass().getClassLoader();
ClassLoader previousCl = Thread.currentThread().getContextClassLoader();
try {
// Spark 4 resolves classic.SparkSession via the context classloader /
plugin CL
Thread.currentThread().setContextClassLoader(pluginCl);
+ previousNamespace = HopVfsNamespaces.bindThread(vfsNamespace);
Review Comment:
**Blocker:** this only works for the first Spark run in the JVM. The Spark
task threads get the namespace by inheriting it from the thread that creates
the SparkContext. The next run reuses that session, because nothing calls
`cleanup()` on Hop Server or hop-run. Its tasks still carry the first run's
namespace, which is closed by then, and the path goes back to a local folder.
Repro: `aHopTransformOnSparkWritesThroughTheRunsNamedConnection` run twice
in one JVM, a new `MemoryMetadataProvider` each time, no `cleanup()`. Run 1
gives errors=0. Run 2 logs `Reusing active SparkSession` and gives errors=1
with `Parent directory file:///…/vfs9591: does not exist!`.
Suggested fix: bind the mini-pipeline's own namespace on the task thread in
`HopMapPartitionsFn`. That pipeline already takes a namespace for the
serialized metadata. Text File Output just doesn't find it through its
variables.
```java
// in call(), after `rows` is built
rows.vfsNamespace = HopVfsNamespaces.resolve(pipeline);
// PartitionIterator
HopVfsNamespace vfsNamespace;
@Override
public boolean hasNext() {
HopVfsNamespace previous = HopVfsNamespaces.bindThread(vfsNamespace);
try {
return hasNextBound(); // the current body of hasNext()
} finally {
HopVfsNamespaces.restoreThread(previous);
}
}
```
With this, both runs pass, and the PR's test passes even without the engine
change. It should also cover executors in other JVMs, which never see the
driver's binding (not tested on a cluster). A second run in the test would keep
this covered.
##########
plugins/engines/spark/src/main/java/org/apache/hop/spark/engines/SparkPipelineEngine.java:
##########
@@ -430,6 +448,7 @@ public void run() {
throw new HopException("Unexpected error starting Spark pipeline", e);
Review Comment:
**Should fix:** if `startThreads()` fails before the `hop-spark-pipeline`
thread starts, the namespace taken in `prepareExecution()` is never released. A
wrong master or an unreachable cluster in `createSparkSession()` is enough.
Only that thread and `cleanup()` release it, and Hop Server and hop-run don't
call `cleanup()`. On a server that leaks one file system manager per failed
start. Calling `releaseVfsNamespace()` in this catch fixes it. It already
guards against a second release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]