bamaer commented on PR #9402:
URL: https://github.com/apache/hop/pull/9402#issuecomment-6772774774

   Thanks @rmannibucau for splitting this out. The points from #9374 are 
addressed, and the failed-job accounting fixes are good.
   
   ### Blockers
   
   **1. The 30 s cap also applies to the normal end of input, so slow rows are 
lost.**
   `finish()` now calls the same capped `awaitJobs()` as `dispose()`. When the 
queued jobs need more than 30 s to drain, `finish()` logs "Abandoning…" at 
basic level and calls `setOutputDone()` while workers are still running. Their 
rows reach the output after it is closed. In Language Model Chat, 
`facadeMap.clear()` also runs first, so the next queued job fails.
   
   The default parallelism is 1, so any batch of LLM calls that takes more than 
30 s to finish draining hits this. With a mocked provider that takes 12 s per 
call and 4 input rows:
   
   | | rows when output is marked done | after `dispose()` |
   |---|---|---|
   | `main` | 4 | 4, errors 0 |
   | this PR | 2 (at 30 s) | 3, errors 1 |
   
   The same pattern is in `Html2Text.java:113`, `DetectLanguage.java:119` and 
`StanfordSimpleNlp.java:112`. Those use the shared common pool with up to 
`cores*100` queued jobs, so Stanford parsing in particular can take longer than 
30 s.
   
   Suggested fix: in `finish()`, wait without a cap until the jobs are done or 
the transform is stopped, for example `while (jobs > 0 && !isStopped()) 
lock.wait(1000);`. Keep the 30 s cap for `dispose()` and the stop path only. 
With that change in `LanguageModelChat.java:84`, the test above returns all 4 
rows and `LanguageModelChatLifecycleTest` still passes.
   
   ### Should fix
   
   **2. Execute Process: under `hop-run`, the forced kill never runs.**
   The 30 s grace wait and `destroyForcibly()` now run on a daemon thread, and 
`processRow` returns as soon as the stop is seen. `hop-run` calls 
`System.exit()` when the pipeline ends, which ends that thread before the 
forced kill. A command that ignores SIGTERM keeps running after the pipeline 
has stopped. On `main`, `stopRunning()` waited until the forced kill. My 
earlier suggestion to put this on the cleanup thread missed this case, sorry.
   
   Suggested fix: keep `stopRunning()` non-blocking, but run `destroy()`, the 
grace wait and `destroyForcibly()` on the transform thread before `processRow` 
returns. Only the stream closing goes to the background thread. 
(`ExecProcess.java:331`)
   
   **What was verified:** the code was reviewed and the PR's unit tests were 
run in all five modules (109 tests, all passing, checkstyle on). A scratch test 
reproduced point 1 on the PR and on `main`, and checked the suggested fix. 
Point 2 is from reading the code and was not run. The existing integration 
tests (`0062-execute-process`, `0086-html2text`, `languagemodelchat`) were not 
run.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to