bamaer opened a new issue, #9005:
URL: https://github.com/apache/hop/issues/9005
### Apache Hop version?
2.20.0-SNAPSHOT (current `main`/develop, commit b06f3588b4)
### Java version?
OpenJDK 21.0.12.1
### Operating system
Linux (x86_64)
### What happened?
A plugin that does work in the `PipelineCompleted` extension point loses it
when the process is `hop-run`: the extension point runs *after* the threads
blocked in `Pipeline.waitUntilFinished()` are released, and `HopRun` exits the
JVM as soon as it is released.
In `Pipeline.fireExecutionFinishedListeners()`:
```java
pipelineCompleted(); // releases
waitUntilFinished()
ExtensionPointHandler.callExtensionPoint(..., PipelineCompleted.id, this);
```
`Pipeline.pipelineCompleted()` sets the result and calls `notifyAll()` on
the waiters. `HopRun.run()` is waiting there, prints `HopRun exit.` and calls
`System.exit(0)` immediately, so the extension point never runs. The lineage
hub flushes its queue exactly there (`LineageHubPipelineCompletedXp`), and its
dispatcher is a daemon thread, so the whole queue dies with the JVM.
Measured with the OpenLineage sink, a two-transform pipeline that takes 0.07
s: **0 of 8 events arrive, in 12 of 12 `hop-run` runs.** Keeping the JVM alive
with a 3 s `Delay` transform makes it deliver all 8 — i.e. the sample only
works because it stays alive, not because the events get out.
Two related orderings lose the tail even when the JVM survives:
* **The last transform's `TransformFinished` is registered behind the
pipeline's completion detector.** `handleTransform` adds the completion
detection with `getTransformFinishedListeners().add(0, finishedListener)` and
only then adds the listener that calls the `TransformFinished` extension point.
The detector fires first, so the last transform's `FINISHED` event (and its
I/O) is queued *after* the flush. Measured: with the reorder above plus a flush
before `System.exit`, 11 of 12 runs delivered 8 of 8; moving the
`TransformFinished` registration ahead of the detector made it 12 of 12.
* **The `SingleThreaded` engine never announces its transforms.**
`handleTransform`'s `case SingleThreaded` branch does nothing ("this needs to
be handled by the pipeline executor"), and the executor never calls
`TransformBeforeStart`/`TransformFinished`. That run configuration therefore
emits only the pipeline's own events: 4 of the expected 8 measured, with no
transform lifecycle at all.
**Proposed fix** (a local patch exists, with three regression tests in
`PipelineTest`; the engine suite is green):
1. Call the `PipelineCompleted` extension point *before* releasing the
waiters, and release them in a `finally` so a failing extension point can never
deadlock a caller.
2. Register the transform-finished extension point ahead of the completion
detector, before the transform threads start.
3. Register `TransformBeforeStart`/`TransformFinished` for the
`SingleThreaded` run configuration as well.
4. As belt and braces, flush the lineage hub once more in `HopRun` before
`System.exit` (a no-op when lineage is disabled).
Happy to open a PR if that is the preferred route; filed as a bug first
because item 1 changes the order in which existing plugins see
`PipelineCompleted` and `waitUntilFinished()`.
### Issue Priority
High (a documented extension point is unreliable for every plugin that uses
it, and the failure is silent)
### Issue Component
Engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
