bamaer opened a new issue, #9417:
URL: https://github.com/apache/hop/issues/9417

   ### What would you like to happen?
   
   ## What's needed
   
   When a pipeline or workflow fails, Hop records *that* it failed 
(`ExecutionState.failed`, error counts, `Result`), but not *why* in a form a 
tool can use. The cause exists only as log text. That makes failures hard to 
monitor, classify, alert on or act on automatically.
   
   Specific gaps:
   
   - No extension point fires when a transform, action, pipeline or workflow 
fails.
   - `ExecutionState` has no error message, exception, failing component or 
plugin id.
   - Transform init failures are not written to the execution information 
location, and `TransformInitThread` does not set the error count.
   - `errorDescription` in the Hop Server pipeline status is never filled.
   - `hop-run` has no machine-readable result.
   
   ## Proposal
   
   Add a structured failure report, without changing existing behaviour:
   
   - An `ExecutionFailure` object: run and parent chain, failing transform or 
action (name, plugin id, copy), phase (prepare, init, run, finish), exception 
and root cause, SQLState/vendor code, a rule-based category (connectivity, 
authentication, schema mismatch, data conversion, constraint violation, 
resource, configuration, child failure, unknown), input/output row layout, 
counters and a redacted log excerpt.
   - New extension points `TransformFailed`, `ActionFailed`, `PipelineFailed` 
and `WorkflowFailed`, plus a listener interface.
   - An optional `failure` field on `ExecutionState`, so every execution 
information location and `/hop/api/v1` exposes it.
   - Register the execution before init, so init failures get a record.
   - Fill `errorDescription` in the Hop Server status.
   - Opt-in `hop-run --result-json <file>`.
   - One core redactor for secrets in failure data.
   
   ## Compatibility
   
   All additions are optional. Older Hop versions reading newer execution 
records keep working for the file, remote and database locations. The Elastic 
location reads with a strict JSON mapper, which needs a fix.
   
   ## Out of scope
   
   - Capturing offending rows
   - Beam and native Spark
   - Automatic retries or other actions on a failure
   - The `hop run` exit code (separate issue)
   
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Actions, Component: Workflows, Component: Transforms, Component: 
Pipelines, Component: Metadata


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to