bamaer commented on PR #9402: URL: https://github.com/apache/hop/pull/9402#issuecomment-6772774774
Thanks @rmannibucau for splitting this out. The points from #9374 are addressed, and the failed-job accounting fixes are good. ### Blockers **1. The 30 s cap also applies to the normal end of input, so slow rows are lost.** `finish()` now calls the same capped `awaitJobs()` as `dispose()`. When the queued jobs need more than 30 s to drain, `finish()` logs "Abandoning…" at basic level and calls `setOutputDone()` while workers are still running. Their rows reach the output after it is closed. In Language Model Chat, `facadeMap.clear()` also runs first, so the next queued job fails. The default parallelism is 1, so any batch of LLM calls that takes more than 30 s to finish draining hits this. With a mocked provider that takes 12 s per call and 4 input rows: | | rows when output is marked done | after `dispose()` | |---|---|---| | `main` | 4 | 4, errors 0 | | this PR | 2 (at 30 s) | 3, errors 1 | The same pattern is in `Html2Text.java:113`, `DetectLanguage.java:119` and `StanfordSimpleNlp.java:112`. Those use the shared common pool with up to `cores*100` queued jobs, so Stanford parsing in particular can take longer than 30 s. Suggested fix: in `finish()`, wait without a cap until the jobs are done or the transform is stopped, for example `while (jobs > 0 && !isStopped()) lock.wait(1000);`. Keep the 30 s cap for `dispose()` and the stop path only. With that change in `LanguageModelChat.java:84`, the test above returns all 4 rows and `LanguageModelChatLifecycleTest` still passes. ### Should fix **2. Execute Process: under `hop-run`, the forced kill never runs.** The 30 s grace wait and `destroyForcibly()` now run on a daemon thread, and `processRow` returns as soon as the stop is seen. `hop-run` calls `System.exit()` when the pipeline ends, which ends that thread before the forced kill. A command that ignores SIGTERM keeps running after the pipeline has stopped. On `main`, `stopRunning()` waited until the forced kill. My earlier suggestion to put this on the cleanup thread missed this case, sorry. Suggested fix: keep `stopRunning()` non-blocking, but run `destroy()`, the grace wait and `destroyForcibly()` on the transform thread before `processRow` returns. Only the stream closing goes to the background thread. (`ExecProcess.java:331`) **What was verified:** the code was reviewed and the PR's unit tests were run in all five modules (109 tests, all passing, checkstyle on). A scratch test reproduced point 1 on the PR and on `main`, and checked the suggested fix. Point 2 is from reading the code and was not run. The existing integration tests (`0062-execute-process`, `0086-html2text`, `languagemodelchat`) were not run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
