+1 for this. From an ML/LLM-Ops standpoint, the introduction of a "human in the loop" operator appears exceptionally valuable. I can envision this operator being effectively integrated into ML pipelines, allowing for dynamic adjustments to model training parameters based on user feedback received during the workflow execution.
Furthermore, the utility of such an operator extends beyond ML applications. There are numerous use cases within standard data and infrastructure workflows where this functionality could be beneficial, such as implementing data quality overrides, securing compliance sign-offs, or even pausing incident response pipelines for human intervention. Best regards, *Ankit Chaurasia* On Wed, May 21, 2025 at 12:42 PM Aritra Basu <aritrabasu1...@gmail.com> wrote: > I'm +0 for it, I like the broad strokes of the idea but actual experience > will very much depend on how we implement it. > > I do very much agree for it to have api support and if it would be good if > we can design it with some thought to what future extensions might be > needed on this feature. > -- > Regards, > Aritra Basu > > On Wed, 21 May 2025, 11:51 am Amogh Desai, <amoghdesai....@gmail.com> > wrote: > > > I really like the idea and see it to be a window for some AI use cases to > > be onboarded with > > the orchestration capabilities of Airflow. > > > > I also concur with what Jens mentioned, it should be an opt-in / opt-out > > feature without > > impacting how Airflow works with schedule today. > > > > As I see it, it's a special case of "event" driven scheduling where the > > "event" is a human input. > > We should clearly define what happens to the task or the dag run if the > > human responsible for > > the input doesn't respond in a timely manner. Either escalate, fail, or > > make an educated guess > > would be some of the thoughts I have in mind. > > > > I really like the idea and the inputs from others here and I would love > to > > join the party for this. > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Wed, May 21, 2025 at 10:07 AM Avi <a...@astronomer.io.invalid> wrote: > > > > > +1 for the opt-in feature. Can clearly see the need for this to exist. > > And > > > for both UI and APIs. +1 with Julian's idea of Dags waiting for input. > > > > > > Thanks, > > > - Avi > > > > > > On Wed, May 21, 2025 at 3:42 AM Kalyan Reddy <kaly...@apache.org> > wrote: > > > > > > > I'm +1 on the idea. There should probably be notifications when > > approval > > > > is needed to continue a task. Additionally, a dashboard would be > > helpful > > > > for viewing the tasks awaiting my approval. It kind of makes me > > remember > > > > approval gates on Azure DevOps pipelines. Even integrations to > > > > Slack/PagerDuty to get alerts and approve/deny externally sound > > > > interesting. > > > > > > > > Thanks, > > > > Kalyan R > > > > > > > > On 2025/05/21 02:07:55 Wei Lee wrote: > > > > > I like this idea. As Jens mentioned, this should be an opt-in > > feature, > > > > and starting with providers seems reasonable. After that, we could > > > probably > > > > get feedback from users and see how we could improve UI/UX. > > > > > > > > > > Best, > > > > > Wei > > > > > > > > > > > On May 21, 2025, at 5:36 AM, Jens Scheffler > > > <j_scheff...@gmx.de.INVALID> > > > > wrote: > > > > > > > > > > > > Hi All, > > > > > > > > > > > > oh I am a bit "envy" on receiving the email because I am thinking > > > > about EXACTLY the same propsal but did not file an AIP to discuss > about > > > > this... as a matter of personal capacity and I wanted to finish off > > > > existing obligations before dropping new ideas. > > > > > > > > > > > > The very same idea for me was called "Human Interaction" provider > > > > package (maybe shot name "apache-airflow-providers-human" with a set > of > > > > operators/sensors to be integrated in a DAG (if needed) to > > Approve/Rejct > > > > ==ShortCircuit Operator, an Operator to select by Human as a > > > > "HumanBranchOprator" and also if data is missing allowing to request > > more > > > > data with the TriggerForm component. > > > > > > > > > > > > So nobody is forcing to have this but it can be easily a provider > > > > package w/o need to change main parts in core. Everybod is free to > > decide > > > > whether it is a good thing to "block" (with a sensor) a workflow to > > > approve > > > > - for example you make a Terraform Plan and sombody need to double > > check > > > > before you Terraform Apply. Maybe you run an AI traning set and if > the > > > > threshold of data is exceeded somebody need to approve budget. Or you > > > build > > > > an AI model and based on the metrics somebody need to approve > > deployment. > > > > Many cases possible. > > > > > > > > > > > > I was also a bit reluctant to push this topic now (myself... but > > now > > > > the idea is un-bottled :-D) bcause I assume AIP-68 would need to be > > > > implemented first with allowing the UI to extnd with UI widgets in > > React > > > > allowing the human forms being seamlessly integrated in the new UI. > So > > I > > > > see AIP-68 as a dependency. > > > > > > > > > > > > I think it might be good to document details on an AIP to align > > idas > > > > and scope. Would like to join the party. (with my currentl limited > > > capacity > > > > that aims to complete more tasks than opening more) > > > > > > > > > > > > Jens > > > > > > > > > > > > On 20.05.25 22:02, Constance Martineau wrote: > > > > > >> I like the idea! We built something similar where a task would > > send > > > an > > > > > >> email with a report attached (Excel, of course). The user had to > > > > either > > > > > >> approve the report or flag an issue. If they didn't respond in 2 > > > > hours, it > > > > > >> escalated to a backup, then to the department lead. If no one > > > > responded, > > > > > >> the DAG failed. It was pretty effective for business-critical > > > > approvals, > > > > > >> and we had clear timeouts and fallback logic. > > > > > >> > > > > > >> That said, I think this pattern works best when it's used > > sparingly > > > > and > > > > > >> with good defaults. Echoing Alexander, human input is inherently > > > > > >> unreliable, but sometimes you really do it, whether it's for > > > > validating a > > > > > >> data quality issue, resolving a content flag, or signing off on > > > > something > > > > > >> before moving forward. I don't think that breaks the Airflow > model > > > as > > > > long > > > > > >> as: > > > > > >> > > > > > >> - You can define what happens if no one responds (timeout, > > fail, > > > > > >> escalate, etc.) > > > > > >> - There's an API so you're not forcing people into the UI > > > > > >> - There's a centralized view of all "waiting for input" tasks > > to > > > > avoid > > > > > >> hunting them down > > > > > >> > > > > > >> It shouldn't become a crutch for manual processes, but I'd love > to > > > see > > > > > >> first-class support for this kind of interaction. It definitely > > > comes > > > > up in > > > > > >> the wild. > > > > > >> > > > > > >> On Tue, May 20, 2025 at 3:46 PM Alexander Shorin < > > kxe...@apache.org > > > > > > > > wrote: > > > > > >> > > > > > >>> While this is an understandable idea, I think it's wrong to > > involve > > > > humans > > > > > >>> to take a part into Airflow workflow. > > > > > >>> Run with parameters. Read the logs. Retry. > > > > > >>> Blocking the whole pipeline could not be acceptable: humans are > > not > > > > stable > > > > > >>> resource for input, they may be sick or on vacation and who > will > > be > > > > > >>> operator for this branch? > > > > > >>> Better strategy to collect metrics and logs and do post-review > of > > > > tasks > > > > > >>> results. > > > > > >>> > > > > > >>> > > > > > >>> -- > > > > > >>> ,,,^..^,,, > > > > > >>> > > > > > >>> On Tue, May 20, 2025 at 10:15 PM Vikram Koka > > > > <vik...@astronomer.io.invalid > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hey everyone, > > > > > >>>> > > > > > >>>> This is a small technical change, but conceptually it is > > > > significant and > > > > > >>>> therefore bringing it to the devlist. > > > > > >>>> > > > > > >>>> As we have been having conversations with Airflow users, who > are > > > > using > > > > > >>>> Airflow for Gen AI applications, there are a couple of > features > > > > that have > > > > > >>>> been brought up as being desirable. As context, based on the > > most > > > > recent > > > > > >>>> Airflow survey, over 8% of Airflow users are now using Airflow > > for > > > > Gen AI > > > > > >>>> use cases. > > > > > >>>> > > > > > >>>> One of the desired features is to have "human interaction" > > within > > > > the > > > > > >>>> context of the overall orchestration flow. Here are the most > > > common > > > > > >>> flows: > > > > > >>>> 1. Choose a branch in the workflow: This is like a "human > > > branch > > > > > >>>> operator", where the human gets to pick one of the paths to > > > move > > > > > >>>> forward. > > > > > >>>> This is a common pattern for business use cases such as > > content > > > > > >>>> moderation, > > > > > >>>> when the automated sentiment analysis is unsure if the > > content > > > is > > > > > >>>> "appropriate" and requires human judgement. > > > > > >>>> 2. Approve / Reject: This is a specialized form of the > above, > > > > where > > > > > >>> the > > > > > >>>> human approves or rejects the output from the prior tasks, > > > > thereby > > > > > >>>> triggering a different branch in the flow. > > > > > >>>> 3. Provide input: The human is expected to provide textual > > > input, > > > > > >>> which > > > > > >>>> can then be used for subsequent steps. This is intended to > be > > > > used > > > > > >>>> within > > > > > >>>> LLM workflows, where a human supplied context / guidance > may > > be > > > > > >>>> critical. > > > > > >>>> > > > > > >>>> All of these above operations are expected to be performed > > through > > > > the > > > > > >>>> Airflow UI. > > > > > >>>> > > > > > >>>> 4. A distinct variation of the above could be an API > interaction > > > > for the > > > > > >>>> above workflows without using the UI. > > > > > >>>> > > > > > >>>> > > > > > >>>> Technical notes: > > > > > >>>> a. While the above tasks are being run (or rather waiting for > > > human > > > > > >>> input), > > > > > >>>> the DAG will be treated as a continued DAG run. > > > > > >>>> > > > > > >>>> b. The Airflow UI for these tasks would be a standard UI form > > > > similar to > > > > > >>>> the Trigger DAG form. > > > > > >>>> For flows (1) and (2) above, this would be populated based on > > the > > > > > >>> follow-on > > > > > >>>> task options. > > > > > >>>> For flow (3) above, this would be a UI form similar to DAG > > params. > > > > > >>>> > > > > > >>>> c. Each of the above operations will be treated as a "task". > > > > > >>>> The task state while waiting for human input will probably be > a > > > > variation > > > > > >>>> of the "deferred task state". TBD as we get closer to > > > > implementation. > > > > > >>>> For flow (3), the input data will be passed using XCom to > > > subsequent > > > > > >>> tasks. > > > > > >>>> > > > > > >>>> As detailed above, the technical changes are fairly small. Any > > > > feedback > > > > > >>> is > > > > > >>>> welcome. > > > > > >>>> > > > > > >>>> Best regards, > > > > > >>>> Vikram > > > > > >>>> -- > > > > > >>>> > > > > > >>>> Vikram Koka > > > > > >>>> Chief Strategy Officer > > > > > >>>> Email: vik...@astronomer.io > > > > > >>>> > > > > > >>>> > > > > > >>>> <https://www.astronomer.io/> > > > > > >>>> > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > >
