It's a bit of a hack, but to save up slots you could just have an instantly-failing PythonOperator (just raise an exception in the callable) that would go in a failed state. Marking it as "success" when the conditions are met would act as a trigger.
On Fri, Oct 5, 2018 at 9:07 AM Brian Greene <br...@heisenbergwoodworking.com> wrote: > My first thought was this, but my understanding is That if you had a > large number of dags “waiting” the sensor would consume all the concurrency. > > And what if the user doesn’t approve? > > How about the dag you have as it’s last step writes to an api/db the > status. > > Then 2 other dags (or one with a branch) can each have a sensor that’s > watching for approved/unapproved values. When it finds one (or a batch > depending on how you write it), trigger the “next” dag. > > This leaves only 1-2 sensors running and would enable your process without > anyone using the airflow UI (assuming they have some other way to mark > “approval”). This avoids the “process by error and recover” logic it seems > like you’d like to get out of. (Which makes sense to me) > > B > > Sent from a device with less than stellar autocorrect > > > On Oct 4, 2018, at 10:17 AM, Alek Storm <alek.st...@gmail.com> wrote: > > > > Hi Björn, > > > > We also sometimes require manual validation, and though we haven't yet > > implemented this, I imagine you could store the approved/unapproved > status > > of the job in a database, expose it via an API, and write an Airflow > sensor > > that continuously polls that API until the status becomes "approved", at > > which point the DAG execution will continue. > > > > Best, > > Alek Storm > > > > On Thu, Oct 4, 2018 at 10:05 AM Björn Pollex > > <bjoern.pol...@soundcloud.com.invalid> wrote: > > > >> Hi all, > >> > >> In some of our workflows we require a manual validation step, where some > >> generated data has to be reviewed by an authorised person before the > >> workflow can continue. We currently model this by using a custom dummy > >> operator that always fails. After the review, we manually mark it as > >> success and clear the downstream tasks. This works, but it would be > nice to > >> have better representation of this in the UI. The customisation points > for > >> plugins don’t seem to offer any way of customising UI for specific > >> operators. > >> > >> Does anyone else have similar use cases? How are you handling this? > >> > >> Cheers, > >> > >> Björn Pollex > >> > >> >
