+1. Sounds pretty useful to have it in Airflow. On Wed, May 21, 2025 at 4:24 PM Vincent Beck <vincb...@apache.org> wrote:
> +1 as well. I can see this feature useful in many different cases (and not > only AI/ML related in my opinion). Happy to read the AIP and give feedbacks > :) > > On 2025/05/21 07:37:01 Tom Rutter wrote: > > +1 here!. As part of our Airflow deployment, we have built a layer on top > > to do review/approve and branching. It would be great for more of this to > > be standardised (and managed in the community by developers who are more > > skilled than me!). > > > > On Wed, 21 May 2025 at 09:12, Ankit Chaurasia <sunank...@gmail.com> > wrote: > > > > > +1 for this. From an ML/LLM-Ops standpoint, the introduction of a > "human in > > > the loop" operator appears exceptionally valuable. I can envision this > > > operator being effectively integrated into ML pipelines, allowing for > > > dynamic adjustments to model training parameters based on user feedback > > > received during the workflow execution. > > > > > > Furthermore, the utility of such an operator extends beyond ML > > > applications. There are numerous use cases within standard data and > > > infrastructure workflows where this functionality could be beneficial, > such > > > as implementing data quality overrides, securing compliance sign-offs, > or > > > even pausing incident response pipelines for human intervention. > > > > > > Best regards, > > > *Ankit Chaurasia* > > > > > > > > > > > > > > > > > > > > > On Wed, May 21, 2025 at 12:42 PM Aritra Basu <aritrabasu1...@gmail.com > > > > > wrote: > > > > > > > I'm +0 for it, I like the broad strokes of the idea but actual > experience > > > > will very much depend on how we implement it. > > > > > > > > I do very much agree for it to have api support and if it would be > good > > > if > > > > we can design it with some thought to what future extensions might be > > > > needed on this feature. > > > > -- > > > > Regards, > > > > Aritra Basu > > > > > > > > On Wed, 21 May 2025, 11:51 am Amogh Desai, <amoghdesai....@gmail.com > > > > > > wrote: > > > > > > > > > I really like the idea and see it to be a window for some AI use > cases > > > to > > > > > be onboarded with > > > > > the orchestration capabilities of Airflow. > > > > > > > > > > I also concur with what Jens mentioned, it should be an opt-in / > > > opt-out > > > > > feature without > > > > > impacting how Airflow works with schedule today. > > > > > > > > > > As I see it, it's a special case of "event" driven scheduling > where the > > > > > "event" is a human input. > > > > > We should clearly define what happens to the task or the dag run > if the > > > > > human responsible for > > > > > the input doesn't respond in a timely manner. Either escalate, > fail, or > > > > > make an educated guess > > > > > would be some of the thoughts I have in mind. > > > > > > > > > > I really like the idea and the inputs from others here and I would > love > > > > to > > > > > join the party for this. > > > > > > > > > > Thanks & Regards, > > > > > Amogh Desai > > > > > > > > > > > > > > > On Wed, May 21, 2025 at 10:07 AM Avi <a...@astronomer.io.invalid> > > > wrote: > > > > > > > > > > > +1 for the opt-in feature. Can clearly see the need for this to > > > exist. > > > > > And > > > > > > for both UI and APIs. +1 with Julian's idea of Dags waiting for > > > input. > > > > > > > > > > > > Thanks, > > > > > > - Avi > > > > > > > > > > > > On Wed, May 21, 2025 at 3:42 AM Kalyan Reddy <kaly...@apache.org > > > > > > wrote: > > > > > > > > > > > > > I'm +1 on the idea. There should probably be notifications when > > > > > approval > > > > > > > is needed to continue a task. Additionally, a dashboard would > be > > > > > helpful > > > > > > > for viewing the tasks awaiting my approval. It kind of makes me > > > > > remember > > > > > > > approval gates on Azure DevOps pipelines. Even integrations to > > > > > > > Slack/PagerDuty to get alerts and approve/deny externally sound > > > > > > > interesting. > > > > > > > > > > > > > > Thanks, > > > > > > > Kalyan R > > > > > > > > > > > > > > On 2025/05/21 02:07:55 Wei Lee wrote: > > > > > > > > I like this idea. As Jens mentioned, this should be an opt-in > > > > > feature, > > > > > > > and starting with providers seems reasonable. After that, we > could > > > > > > probably > > > > > > > get feedback from users and see how we could improve UI/UX. > > > > > > > > > > > > > > > > Best, > > > > > > > > Wei > > > > > > > > > > > > > > > > > On May 21, 2025, at 5:36 AM, Jens Scheffler > > > > > > <j_scheff...@gmx.de.INVALID> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > oh I am a bit "envy" on receiving the email because I am > > > thinking > > > > > > > about EXACTLY the same propsal but did not file an AIP to > discuss > > > > about > > > > > > > this... as a matter of personal capacity and I wanted to > finish off > > > > > > > existing obligations before dropping new ideas. > > > > > > > > > > > > > > > > > > The very same idea for me was called "Human Interaction" > > > provider > > > > > > > package (maybe shot name "apache-airflow-providers-human" with > a > > > set > > > > of > > > > > > > operators/sensors to be integrated in a DAG (if needed) to > > > > > Approve/Rejct > > > > > > > ==ShortCircuit Operator, an Operator to select by Human as a > > > > > > > "HumanBranchOprator" and also if data is missing allowing to > > > request > > > > > more > > > > > > > data with the TriggerForm component. > > > > > > > > > > > > > > > > > > So nobody is forcing to have this but it can be easily a > > > provider > > > > > > > package w/o need to change main parts in core. Everybod is > free to > > > > > decide > > > > > > > whether it is a good thing to "block" (with a sensor) a > workflow to > > > > > > approve > > > > > > > - for example you make a Terraform Plan and sombody need to > double > > > > > check > > > > > > > before you Terraform Apply. Maybe you run an AI traning set > and if > > > > the > > > > > > > threshold of data is exceeded somebody need to approve budget. > Or > > > you > > > > > > build > > > > > > > an AI model and based on the metrics somebody need to approve > > > > > deployment. > > > > > > > Many cases possible. > > > > > > > > > > > > > > > > > > I was also a bit reluctant to push this topic now > (myself... > > > but > > > > > now > > > > > > > the idea is un-bottled :-D) bcause I assume AIP-68 would need > to be > > > > > > > implemented first with allowing the UI to extnd with UI > widgets in > > > > > React > > > > > > > allowing the human forms being seamlessly integrated in the > new UI. > > > > So > > > > > I > > > > > > > see AIP-68 as a dependency. > > > > > > > > > > > > > > > > > > I think it might be good to document details on an AIP to > align > > > > > idas > > > > > > > and scope. Would like to join the party. (with my currentl > limited > > > > > > capacity > > > > > > > that aims to complete more tasks than opening more) > > > > > > > > > > > > > > > > > > Jens > > > > > > > > > > > > > > > > > > On 20.05.25 22:02, Constance Martineau wrote: > > > > > > > > >> I like the idea! We built something similar where a task > would > > > > > send > > > > > > an > > > > > > > > >> email with a report attached (Excel, of course). The user > had > > > to > > > > > > > either > > > > > > > > >> approve the report or flag an issue. If they didn't > respond > > > in 2 > > > > > > > hours, it > > > > > > > > >> escalated to a backup, then to the department lead. If no > one > > > > > > > responded, > > > > > > > > >> the DAG failed. It was pretty effective for > business-critical > > > > > > > approvals, > > > > > > > > >> and we had clear timeouts and fallback logic. > > > > > > > > >> > > > > > > > > >> That said, I think this pattern works best when it's used > > > > > sparingly > > > > > > > and > > > > > > > > >> with good defaults. Echoing Alexander, human input is > > > inherently > > > > > > > > >> unreliable, but sometimes you really do it, whether it's > for > > > > > > > validating a > > > > > > > > >> data quality issue, resolving a content flag, or signing > off > > > on > > > > > > > something > > > > > > > > >> before moving forward. I don't think that breaks the > Airflow > > > > model > > > > > > as > > > > > > > long > > > > > > > > >> as: > > > > > > > > >> > > > > > > > > >> - You can define what happens if no one responds > (timeout, > > > > > fail, > > > > > > > > >> escalate, etc.) > > > > > > > > >> - There's an API so you're not forcing people into the > UI > > > > > > > > >> - There's a centralized view of all "waiting for input" > > > tasks > > > > > to > > > > > > > avoid > > > > > > > > >> hunting them down > > > > > > > > >> > > > > > > > > >> It shouldn't become a crutch for manual processes, but I'd > > > love > > > > to > > > > > > see > > > > > > > > >> first-class support for this kind of interaction. It > > > definitely > > > > > > comes > > > > > > > up in > > > > > > > > >> the wild. > > > > > > > > >> > > > > > > > > >> On Tue, May 20, 2025 at 3:46 PM Alexander Shorin < > > > > > kxe...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > >> > > > > > > > > >>> While this is an understandable idea, I think it's wrong > to > > > > > involve > > > > > > > humans > > > > > > > > >>> to take a part into Airflow workflow. > > > > > > > > >>> Run with parameters. Read the logs. Retry. > > > > > > > > >>> Blocking the whole pipeline could not be acceptable: > humans > > > are > > > > > not > > > > > > > stable > > > > > > > > >>> resource for input, they may be sick or on vacation and > who > > > > will > > > > > be > > > > > > > > >>> operator for this branch? > > > > > > > > >>> Better strategy to collect metrics and logs and do > > > post-review > > > > of > > > > > > > tasks > > > > > > > > >>> results. > > > > > > > > >>> > > > > > > > > >>> > > > > > > > > >>> -- > > > > > > > > >>> ,,,^..^,,, > > > > > > > > >>> > > > > > > > > >>> On Tue, May 20, 2025 at 10:15 PM Vikram Koka > > > > > > > <vik...@astronomer.io.invalid > > > > > > > > >>> wrote: > > > > > > > > >>> > > > > > > > > >>>> Hey everyone, > > > > > > > > >>>> > > > > > > > > >>>> This is a small technical change, but conceptually it is > > > > > > > significant and > > > > > > > > >>>> therefore bringing it to the devlist. > > > > > > > > >>>> > > > > > > > > >>>> As we have been having conversations with Airflow > users, who > > > > are > > > > > > > using > > > > > > > > >>>> Airflow for Gen AI applications, there are a couple of > > > > features > > > > > > > that have > > > > > > > > >>>> been brought up as being desirable. As context, based > on the > > > > > most > > > > > > > recent > > > > > > > > >>>> Airflow survey, over 8% of Airflow users are now using > > > Airflow > > > > > for > > > > > > > Gen AI > > > > > > > > >>>> use cases. > > > > > > > > >>>> > > > > > > > > >>>> One of the desired features is to have "human > interaction" > > > > > within > > > > > > > the > > > > > > > > >>>> context of the overall orchestration flow. Here are the > most > > > > > > common > > > > > > > > >>> flows: > > > > > > > > >>>> 1. Choose a branch in the workflow: This is like a > "human > > > > > > branch > > > > > > > > >>>> operator", where the human gets to pick one of the > paths > > > to > > > > > > move > > > > > > > > >>>> forward. > > > > > > > > >>>> This is a common pattern for business use cases such > as > > > > > content > > > > > > > > >>>> moderation, > > > > > > > > >>>> when the automated sentiment analysis is unsure if > the > > > > > content > > > > > > is > > > > > > > > >>>> "appropriate" and requires human judgement. > > > > > > > > >>>> 2. Approve / Reject: This is a specialized form of > the > > > > above, > > > > > > > where > > > > > > > > >>> the > > > > > > > > >>>> human approves or rejects the output from the prior > > > tasks, > > > > > > > thereby > > > > > > > > >>>> triggering a different branch in the flow. > > > > > > > > >>>> 3. Provide input: The human is expected to provide > > > textual > > > > > > input, > > > > > > > > >>> which > > > > > > > > >>>> can then be used for subsequent steps. This is > intended > > > to > > > > be > > > > > > > used > > > > > > > > >>>> within > > > > > > > > >>>> LLM workflows, where a human supplied context / > guidance > > > > may > > > > > be > > > > > > > > >>>> critical. > > > > > > > > >>>> > > > > > > > > >>>> All of these above operations are expected to be > performed > > > > > through > > > > > > > the > > > > > > > > >>>> Airflow UI. > > > > > > > > >>>> > > > > > > > > >>>> 4. A distinct variation of the above could be an API > > > > interaction > > > > > > > for the > > > > > > > > >>>> above workflows without using the UI. > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> Technical notes: > > > > > > > > >>>> a. While the above tasks are being run (or rather > waiting > > > for > > > > > > human > > > > > > > > >>> input), > > > > > > > > >>>> the DAG will be treated as a continued DAG run. > > > > > > > > >>>> > > > > > > > > >>>> b. The Airflow UI for these tasks would be a standard UI > > > form > > > > > > > similar to > > > > > > > > >>>> the Trigger DAG form. > > > > > > > > >>>> For flows (1) and (2) above, this would be populated > based > > > on > > > > > the > > > > > > > > >>> follow-on > > > > > > > > >>>> task options. > > > > > > > > >>>> For flow (3) above, this would be a UI form similar to > DAG > > > > > params. > > > > > > > > >>>> > > > > > > > > >>>> c. Each of the above operations will be treated as a > "task". > > > > > > > > >>>> The task state while waiting for human input will > probably > > > be > > > > a > > > > > > > variation > > > > > > > > >>>> of the "deferred task state". TBD as we get closer to > > > > > > > implementation. > > > > > > > > >>>> For flow (3), the input data will be passed using XCom > to > > > > > > subsequent > > > > > > > > >>> tasks. > > > > > > > > >>>> > > > > > > > > >>>> As detailed above, the technical changes are fairly > small. > > > Any > > > > > > > feedback > > > > > > > > >>> is > > > > > > > > >>>> welcome. > > > > > > > > >>>> > > > > > > > > >>>> Best regards, > > > > > > > > >>>> Vikram > > > > > > > > >>>> -- > > > > > > > > >>>> > > > > > > > > >>>> Vikram Koka > > > > > > > > >>>> Chief Strategy Officer > > > > > > > > >>>> Email: vik...@astronomer.io > > > > > > > > >>>> > > > > > > > > >>>> > > > > > > > > >>>> <https://www.astronomer.io/> > > > > > > > > >>>> > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > > > > For additional commands, e-mail: > dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > > > > > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
