bamaer opened a new issue, #9631:
URL: https://github.com/apache/hop/issues/9631

   ### What would you like to happen?
   
   A workflow engine that runs a Hop workflow as an Apache Airflow DAG. 
Workflows are still designed in Hop GUI; Airflow handles scheduling, retries 
and per-action task state.
   
   - **Translator:** builds a DAG from each `.hwf` with Airflow's Java Task SDK 
(Airflow 3.4+). The Hop project is the DAG bundle, and Hop is installed on the 
workers.
   - **Fragment runner:** runs one action, or a group of actions, in a task. 
Variables and the previous result are handed over through XCom; the runner 
returns the result, the changed variables and the result rows. Loops, actions 
that would run more than once, and "Execute for every input row" run as one 
fragment, so results don't change.
   - **"Airflow" run configuration in Hop GUI:** publish, trigger, follow task 
state on the canvas, and stop. It uses `airflow-connection` from #9630. The 
engine's `supports()` marks the actions it can't run.
   
   On a sample of real-world workflows, about 9 in 10 map one action per task, 
but this depends on a project's style. The rest run partly as Hop fragments. 
The goal is that every workflow runs with the same results as on the local 
engine. About a quarter of workflows pass state between actions, so the XCom 
handover is needed from the start.
   
   The Java SDK reached 1.0.0 in November 2026 and is still experimental. It 
has no DAG params, which matters because many workflows have parameters, and it 
doesn't enforce `execution_timeout` yet. Keep it behind one adapter module and 
pin its version per Hop release. Start with a spike that runs one task per 
workflow and covers passing parameters, then move to one task per action.
   
   ### Issue Priority
   
   Priority: 3
   
   ### Issue Component
   
   Component: Workflows
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to