rmannibucau opened a new pull request, #9481:
URL: https://github.com/apache/hop/pull/9481

   ## Summary
   
   Introduce a generic, opt-in execution observation contract for local 
pipelines and workflows. Execution lifetime is separate from thread-local 
activation, allowing integrations to observe complete invocations without 
introducing a telemetry SDK dependency into Hop.
   
   - Add execution handles, kinds, terminal states, observers and closeable 
activation scopes.
   - Isolate observer failures from business execution; preserve existing 
extension-point semantics.
   - Observe actual transform copies and action clones, including 
initialization, processing and disposal.
   - Capture the actual launching action/transform before asynchronous child 
dispatch in ActionPipeline, ActionWorkflow, PipelineExecutor and 
WorkflowExecutor.
   - Support local multithreaded and single-threaded execution, with no per-row 
execution identities or notifications. Without observers, no execution registry 
entries or generated identities are created.
   - Defer pipeline terminal publication until startup and running workers have 
settled; handle fast completion, callback failures and partial worker launch.
   - Add lifecycle, restoration, child-parentage, asynchronous action and 
runner-completion regression tests.
   
   There is no OpenTelemetry, MDC or exporter dependency in the engine changes. 
A separate integration was used to exercise the contract; its plugin code and 
dependencies are not included here.
   
   ## Scope and review points
   
   This is a proposed public API, based on Hop 
`2d995a647a223a53630ef958462628fe00c6c513` (2.21.0-SNAPSHOT, Java 21). 
Maintainer agreement on the API and a complete independent lifecycle review are 
still outstanding.
   
   Execution lifetime and activation are intentionally distinct. A stop request 
is not completion, asynchronous children may outlive their launcher, and legacy 
workflow-runner postludes remain separate from the engine invocation. See 
`docs/otel-execution-design.md`.
   
   Remote execution, HopServer, Beam/Spark workers and arbitrary third-party 
threads are not qualified by this change. Long-duration/production-scale leak 
and performance qualification is also outside the validation performed here.
   
   ## Validation
   
   ### Fresh pre-PR checks
   
   Passed:
   
   ```sh
   mvn -q -pl engine jacoco:prepare-agent surefire:test \
     
-Dtest=ExecutionLifecycleTest,PipelineExecutionLifecycleTest,PipelineTest,WorkflowLifecycleTest,ExecutionFailurePipelineTest,ExecutionFailureWorkflowTest
 \
     -Dspotless.skip=true -Drat.skip=true -DargLine= \
     -DforkedProcessTimeoutInSeconds=180
   
   mvn -q -pl 
engine,plugins/actions/pipeline,plugins/actions/workflow,plugins/transforms/pipelineexecutor,plugins/transforms/workflowexecutor,plugins/engines/single-threaded
 \
     -Dspotless.skip=false spotless:check
   
   git diff --cached --check
   ```
   
   **Known current test failure (reason for draft status):** the launcher-suite 
command below was rerun twice and timed out in 
`ActionWorkflowAsyncLifecycleTest.realNoWaitLauncherCapturesActionBeforeParentEndsAndDoesNotWaitForChild`
 at its 10-second JUnit timeout. The root cause has not been established; this 
is not being claimed as an environment-only failure. The reactor stops in the 
workflow-action module, so subsequent executor modules were not freshly 
revalidated by these attempts.
   
   ```sh
   mvn -q -pl 
plugins/actions/pipeline,plugins/actions/workflow,plugins/transforms/pipelineexecutor,plugins/transforms/workflowexecutor,plugins/engines/single-threaded
 \
     jacoco:prepare-agent surefire:test \
     -Dspotless.skip=true -Drat.skip=true -DargLine= \
     -DforkedProcessTimeoutInSeconds=180
   ```
   
   ### Earlier bounded qualification (historical, not a replacement for the 
failure above)
   
   A previous isolated campaign passed in 281.929 seconds, excluding 
compilation, packaging and downloads. Hop-side results were 73 engine tests, 9 
pipeline-action tests, 5 workflow-action tests, 56 PipelineExecutor tests, 9 
WorkflowExecutor tests and 4 single-threaded-engine tests, with no 
failures/errors/skips in that campaign.
   
   The external integration campaign also exercised distribution ZIPs with an 
OTLP Collector, exact child parentage, 520 repeated real pipeline runs and 
2,600 pipeline/copy invocations, and warmed multithreaded/single-threaded 
benchmarks. These checks do not constitute exhaustive qualification or a stable 
performance guarantee. Their harness is outside this PR.
   
   ## Contribution checklist
   
   - [ ] Run `mvn clean install apache-rat:check` (full reactor/license check 
not run for this PR).
   - [x] Keep the change in one commit.
   - [ ] Mention a corresponding issue, if applicable (none linked).
   - [ ] I hereby declare this contribution to be licensed under the Apache 
License Version 2.0, January 2004 (left for the contributor to acknowledge).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to