There is 4th option to keep things as-is too :) On Wed, 13 Nov 2024 at 12:19, Kaxil Naik <kaxiln...@gmail.com> wrote:
> Hi all, > > Following up on the Dev call discussions last Thursday, I am opening this > up for discussion. > > Reschedule mode was introduced to improve efficiency over poke mode by > allowing tasks to wait without holding a worker slot. Since the > introduction of deferrable operators in Airflow 2.2, however, we now have > an even more optimal, async-driven solution. The adoption of deferrable > operators has been really good, and since we are already chopping things > off with Airflow 3 it might be time to consider making them the default > mode. > > This will ensure that our users always have the most optimal way of > running sensors by default and that we, the maintainers or folks supporting > Airflow deployments in companies, do not need to know different approaches > with Reschedule mode, either. > > However, not all sensors can be async, either due to limitations in > underlying libraries or a lack of unique ids for async polling. > > Knowing that we have a few options: > > 1) *Remove Poke & Reschedule modes* > > This is aggressive and it means we will have to remove things like > PostgresSensor that does not support async. > > 2) *Remove Reschedule mode * > > Make deferrable the primary mode, falling back to poke where async isn’t > supported. > > 3) *Make Deferrable the default, keep Poke & Reschedule* > > This is a defensive option that maintains current behaviour but ensures > that we have the most performant option by default. It could be as simple > as making AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE default to True. > > I’d love to hear feedback, especially from users who rely on reschedule > mode today! > > Regards, > Kaxil >
