While this is an understandable idea, I think it's wrong to involve humans to take a part into Airflow workflow. Run with parameters. Read the logs. Retry. Blocking the whole pipeline could not be acceptable: humans are not stable resource for input, they may be sick or on vacation and who will be operator for this branch? Better strategy to collect metrics and logs and do post-review of tasks results.
-- ,,,^..^,,, On Tue, May 20, 2025 at 10:15 PM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Hey everyone, > > This is a small technical change, but conceptually it is significant and > therefore bringing it to the devlist. > > As we have been having conversations with Airflow users, who are using > Airflow for Gen AI applications, there are a couple of features that have > been brought up as being desirable. As context, based on the most recent > Airflow survey, over 8% of Airflow users are now using Airflow for Gen AI > use cases. > > One of the desired features is to have "human interaction" within the > context of the overall orchestration flow. Here are the most common flows: > > 1. Choose a branch in the workflow: This is like a "human branch > operator", where the human gets to pick one of the paths to move > forward. > This is a common pattern for business use cases such as content > moderation, > when the automated sentiment analysis is unsure if the content is > "appropriate" and requires human judgement. > 2. Approve / Reject: This is a specialized form of the above, where the > human approves or rejects the output from the prior tasks, thereby > triggering a different branch in the flow. > 3. Provide input: The human is expected to provide textual input, which > can then be used for subsequent steps. This is intended to be used > within > LLM workflows, where a human supplied context / guidance may be > critical. > > All of these above operations are expected to be performed through the > Airflow UI. > > 4. A distinct variation of the above could be an API interaction for the > above workflows without using the UI. > > > Technical notes: > a. While the above tasks are being run (or rather waiting for human input), > the DAG will be treated as a continued DAG run. > > b. The Airflow UI for these tasks would be a standard UI form similar to > the Trigger DAG form. > For flows (1) and (2) above, this would be populated based on the follow-on > task options. > For flow (3) above, this would be a UI form similar to DAG params. > > c. Each of the above operations will be treated as a "task". > The task state while waiting for human input will probably be a variation > of the "deferred task state". TBD as we get closer to implementation. > For flow (3), the input data will be passed using XCom to subsequent tasks. > > > As detailed above, the technical changes are fairly small. Any feedback is > welcome. > > Best regards, > Vikram > -- > > Vikram Koka > Chief Strategy Officer > Email: vik...@astronomer.io > > > <https://www.astronomer.io/> >
