hansva opened a new issue, #9526: URL: https://github.com/apache/hop/issues/9526
### Apache Hop version? 2.20.0 ### Java version? 21 ### Operating system Other ### What happened? Three problems in how `SingleThreadedPipelineExecutor` runs transforms. It is used by the single-threaded pipeline engine, the batch sub-pipelines of Kafka consumer and Azure Event Hubs listener, and the Beam engines. 1. **A transform with work left after its input ends is never called again.** `oneIteration()` calls `processRow()` once per row waiting in the transform's input row sets. Once all input has arrived and been read, a transform that still has work gets no more calls. Examples are a reader going through the file a row named, a mapping, or a blocking transform. It never finishes, and its remaining rows are lost. 2. **Simple mapping run single-threaded never finishes.** After its last input row it calls `finished()` on the input mapper and then `waitUntilFinished()` on the sub-pipeline. Nothing drives the iterations of a single-threaded sub-pipeline, so it waits forever. The case without input already drives them (added with #9245). 3. **Table output closes its output on every batch.** `batchComplete()` flushes through `emptyAndCommitBatchBuffers(false)`, whose `finally` calls `setOutputDone()` whatever its argument. Single-threaded execution calls `batchComplete()` after every iteration. From the first iteration that gives a transform after Table output nothing, that transform stops reading. The rows that follow, on the output and on the error hop, are lost. Number 1 is reproduced on the single-threaded engine (steps below). Numbers 2 and 3 were found by running integration tests through the same executor, and can be read in the code on `main`. ### How to reproduce (1) 1. A Data grid with one String field, `filename`, holding the path of a CSV file with a header and 39 data rows. 2. A CSV file input that reads the file named in the field `filename`, with the file's fields. 3. A Text file output. 4. Run with the local engine: 39 rows are written. Run with a run configuration for the single-threaded engine (`LocalSingle`): 1 row is written, without an error. ### Proposed fix 1. `oneIteration()` gives a transform one more `processRow()` call per iteration while its input is finished and empty, until the transform says it is done. 2. Simple mapping drives the iterations of a single-threaded sub-pipeline after its last input row, as it already does without input. 3. Only `dispose()` closes Table output's output. At the end of its rows, `processRow()` writes what is still buffered and closes the output itself. ### Issue Priority Priority: 2 ### Issue Component Component: API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
