GitHub user carloea2 edited a comment on the discussion: Discuss workflow macro design and lifecycle
<img width="1200" height="1600" alt="image" src="https://github.com/user-attachments/assets/a703b82c-22fc-4ce3-b0ee-4e320537c22d"; /> We have been exploring a revised approach to macros and would like to discuss it here before settling the design. Some ideas we are considering: 1. - **Workflow and macro relationship:** At the execution level, workflows and macros are both operator graphs with the same engine capabilities. One possible design choice is to reserve standalone execution for workflows, even though a macro’s DAG may technically be executable independently. 2. - **Composition:** Under this model, a workflow would act as the entry point, similar to `main` in a program. It could contain macros, and macros could contain other macros, similar to functions calling other functions. 3. - **Macro representation:** A macro could represent a sub-DAG as a single operator, with exposed input/output ports connected to its internal graph. 4. - **UI experience:** Reusable macros could appear in the operator panel alongside regular operators. Xiaozhen’s UI could serve as the starting point for editing and navigating macro internals. 5. - **Independent versioning:** We are considering versioning workflows and macros independently, with macro instances referencing their definitions through pointers. For example, workflow A version 8 could reference macro B version 5. 6. - **Execution:** An initial approach could be to recursively expand macros in the frontend before submitting the graph for execution. The saved workflow would retain its hierarchy and references, while the execution engine would receive a flat operator graph. This should avoid engine changes, although persistence and APIs may still need work. This would broaden the earlier proposal that deferred reuse. Whether these capabilities should be included initially or introduced in stages is also open for discussion. Some questions we would like feedback on: 1. **Overall model:** Does the distinction between an executable workflow and a reusable macro make sense? Are there use cases where allowing standalone macro execution would be valuable? 2. **Export/import:** How should a workflow carry its macro dependencies when imported into another deployment? Would bundling the referenced definitions and versions be a suitable approach? 3. **Form view:** How should exposed properties appear across nested macros? Which values should belong to the shared definition, and which should each instance be able to override? 4. **Versioning:** Should references point to specific versions? How should users adopt updates, especially when ports or properties change? These are starting points for discussion. Suggestions for alternative models, missing use cases, or a simpler initial scope would be welcome. GitHub link: https://github.com/apache/texera/discussions/5248#discussioncomment-18788116 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
