I'm +1 on the idea. There should probably be notifications when approval is needed to continue a task. Additionally, a dashboard would be helpful for viewing the tasks awaiting my approval. It kind of makes me remember approval gates on Azure DevOps pipelines. Even integrations to Slack/PagerDuty to get alerts and approve/deny externally sound interesting.
Thanks, Kalyan R On 2025/05/21 02:07:55 Wei Lee wrote: > I like this idea. As Jens mentioned, this should be an opt-in feature, and > starting with providers seems reasonable. After that, we could probably get > feedback from users and see how we could improve UI/UX. > > Best, > Wei > > > On May 21, 2025, at 5:36 AM, Jens Scheffler <j_scheff...@gmx.de.INVALID> > > wrote: > > > > Hi All, > > > > oh I am a bit "envy" on receiving the email because I am thinking about > > EXACTLY the same propsal but did not file an AIP to discuss about this... > > as a matter of personal capacity and I wanted to finish off existing > > obligations before dropping new ideas. > > > > The very same idea for me was called "Human Interaction" provider package > > (maybe shot name "apache-airflow-providers-human" with a set of > > operators/sensors to be integrated in a DAG (if needed) to Approve/Rejct > > ==ShortCircuit Operator, an Operator to select by Human as a > > "HumanBranchOprator" and also if data is missing allowing to request more > > data with the TriggerForm component. > > > > So nobody is forcing to have this but it can be easily a provider package > > w/o need to change main parts in core. Everybod is free to decide whether > > it is a good thing to "block" (with a sensor) a workflow to approve - for > > example you make a Terraform Plan and sombody need to double check before > > you Terraform Apply. Maybe you run an AI traning set and if the threshold > > of data is exceeded somebody need to approve budget. Or you build an AI > > model and based on the metrics somebody need to approve deployment. Many > > cases possible. > > > > I was also a bit reluctant to push this topic now (myself... but now the > > idea is un-bottled :-D) bcause I assume AIP-68 would need to be implemented > > first with allowing the UI to extnd with UI widgets in React allowing the > > human forms being seamlessly integrated in the new UI. So I see AIP-68 as a > > dependency. > > > > I think it might be good to document details on an AIP to align idas and > > scope. Would like to join the party. (with my currentl limited capacity > > that aims to complete more tasks than opening more) > > > > Jens > > > > On 20.05.25 22:02, Constance Martineau wrote: > >> I like the idea! We built something similar where a task would send an > >> email with a report attached (Excel, of course). The user had to either > >> approve the report or flag an issue. If they didn't respond in 2 hours, it > >> escalated to a backup, then to the department lead. If no one responded, > >> the DAG failed. It was pretty effective for business-critical approvals, > >> and we had clear timeouts and fallback logic. > >> > >> That said, I think this pattern works best when it's used sparingly and > >> with good defaults. Echoing Alexander, human input is inherently > >> unreliable, but sometimes you really do it, whether it's for validating a > >> data quality issue, resolving a content flag, or signing off on something > >> before moving forward. I don't think that breaks the Airflow model as long > >> as: > >> > >> - You can define what happens if no one responds (timeout, fail, > >> escalate, etc.) > >> - There's an API so you're not forcing people into the UI > >> - There's a centralized view of all "waiting for input" tasks to avoid > >> hunting them down > >> > >> It shouldn't become a crutch for manual processes, but I'd love to see > >> first-class support for this kind of interaction. It definitely comes up in > >> the wild. > >> > >> On Tue, May 20, 2025 at 3:46 PM Alexander Shorin <kxe...@apache.org> wrote: > >> > >>> While this is an understandable idea, I think it's wrong to involve humans > >>> to take a part into Airflow workflow. > >>> Run with parameters. Read the logs. Retry. > >>> Blocking the whole pipeline could not be acceptable: humans are not stable > >>> resource for input, they may be sick or on vacation and who will be > >>> operator for this branch? > >>> Better strategy to collect metrics and logs and do post-review of tasks > >>> results. > >>> > >>> > >>> -- > >>> ,,,^..^,,, > >>> > >>> On Tue, May 20, 2025 at 10:15 PM Vikram Koka <vik...@astronomer.io.invalid > >>> wrote: > >>> > >>>> Hey everyone, > >>>> > >>>> This is a small technical change, but conceptually it is significant and > >>>> therefore bringing it to the devlist. > >>>> > >>>> As we have been having conversations with Airflow users, who are using > >>>> Airflow for Gen AI applications, there are a couple of features that have > >>>> been brought up as being desirable. As context, based on the most recent > >>>> Airflow survey, over 8% of Airflow users are now using Airflow for Gen AI > >>>> use cases. > >>>> > >>>> One of the desired features is to have "human interaction" within the > >>>> context of the overall orchestration flow. Here are the most common > >>> flows: > >>>> 1. Choose a branch in the workflow: This is like a "human branch > >>>> operator", where the human gets to pick one of the paths to move > >>>> forward. > >>>> This is a common pattern for business use cases such as content > >>>> moderation, > >>>> when the automated sentiment analysis is unsure if the content is > >>>> "appropriate" and requires human judgement. > >>>> 2. Approve / Reject: This is a specialized form of the above, where > >>> the > >>>> human approves or rejects the output from the prior tasks, thereby > >>>> triggering a different branch in the flow. > >>>> 3. Provide input: The human is expected to provide textual input, > >>> which > >>>> can then be used for subsequent steps. This is intended to be used > >>>> within > >>>> LLM workflows, where a human supplied context / guidance may be > >>>> critical. > >>>> > >>>> All of these above operations are expected to be performed through the > >>>> Airflow UI. > >>>> > >>>> 4. A distinct variation of the above could be an API interaction for the > >>>> above workflows without using the UI. > >>>> > >>>> > >>>> Technical notes: > >>>> a. While the above tasks are being run (or rather waiting for human > >>> input), > >>>> the DAG will be treated as a continued DAG run. > >>>> > >>>> b. The Airflow UI for these tasks would be a standard UI form similar to > >>>> the Trigger DAG form. > >>>> For flows (1) and (2) above, this would be populated based on the > >>> follow-on > >>>> task options. > >>>> For flow (3) above, this would be a UI form similar to DAG params. > >>>> > >>>> c. Each of the above operations will be treated as a "task". > >>>> The task state while waiting for human input will probably be a variation > >>>> of the "deferred task state". TBD as we get closer to implementation. > >>>> For flow (3), the input data will be passed using XCom to subsequent > >>> tasks. > >>>> > >>>> As detailed above, the technical changes are fairly small. Any feedback > >>> is > >>>> welcome. > >>>> > >>>> Best regards, > >>>> Vikram > >>>> -- > >>>> > >>>> Vikram Koka > >>>> Chief Strategy Officer > >>>> Email: vik...@astronomer.io > >>>> > >>>> > >>>> <https://www.astronomer.io/> > >>>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
