cool On Fri, Sep 5, 2025 at 8:30 PM Ferruzzi, Dennis <ferru...@amazon.com> wrote:
> Yup, I like it. Thanks for driving this one. > > > - ferruzzi > > > ________________________________ > From: Wei Lee <weilee...@gmail.com> > Sent: Thursday, September 4, 2025 3:43 AM > To: dev@airflow.apache.org > Subject: RE: [EXT] [Discussion] Should We avoid using AirflowException > direclty? > > CAUTION: This email originated from outside of the organization. Do not > click links or open attachments unless you can confirm the sender and know > the content is safe. > > > > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que > le contenu ne présente aucun risque. > > > > I believe we've reached a certain level of consensus, but we still have > some things to discuss. I'll break it into two parts. > > — > > ## What we have concluded on. > 1. In most cases, we should prioritize using Python’s standard exceptions > (e.g., `ValueError`, `TypeError`, `OSError`) instead of wrapping everything > in `AirflowException`. > 2. Within `airflow-core`, we should define and utilize more specific > exception classes under `airflow.exceptions`. > 3. For provider-specific implementations, exceptions should be defined > within `airflow.providers.<provider>.exceptions`. > The use of points 2 and 3 should only be considered when point 1 is > inappropriate, which should be a rare occurrence. > > What we can do other than nail down these rules > Add it to > https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst > Add a prek hook to detect `raise AirflowException` > > If no one objects, I'm planning on sending a [Lazy Consensus] sometime > tomorrow. > > — > > ## What we haven't yet had a consensus > Whether we really can/want to remove or depreciate AirflowException > If we want, how are we going to achieve it > > Best, > Wei > > > On Sep 2, 2025, at 3:36 PM, Amogh Desai <amoghde...@apache.org> wrote: > > > > Generally agree with this too. > > > > `AirflowException` is literally being overused almost to an extent that > it > > has > > become like a dumping ground (along the lines of what `utils` has become) > > > > I would be in favour of removing it too and addressing backcompat > concerns > > for user facing / custom provider / hooks etc facing classes. > > > > This could be run as a community exercise too once we reach that stage to > > clean > > up providers from using / overusing it. > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Tue, Sep 2, 2025 at 1:58 AM Jens Scheffler <j_scheff...@gmx.de.invalid > > > > wrote: > > > >> Also was a bit focussing on other stuff, late to the party. > >> > >> For deprecation we should consider RUFF rules as well. > >> > >> But on side of code changes I fear a lot of people wil have > >> implementations based on AirflowException. I did not get the idea of > >> backwards compatability. If we change Exception types then Operators > >> implemented on top of Standard-Provider might fail if the receive > >> different exceptions from inherited classes? > >> > >> Jens > >> > >> On 01.09.25 17:05, Buğra Öztürk wrote: > >>> Great discussion! Joining delayed. We have this usage in airflowctl too > >>> where AirflowException is the main one even though it isn't imported > from > >>> core. > >>> I agree with the consensus. Great idea to make it backward compatible. > >>> Maybe would be good to define a migration calendar roughly. > >>> While we should prevent usage with some pre-commit hooks, we should > also > >>> document the practices we should follow on exception handling for these > >>> changes. > >>> Even though we can manage to migrate seamlessly for the user, I think > we > >>> still should warn them about the exception handling usage and give some > >>> kind of deprecation warning if possible > >>> > >>> Bugra Ozturk > >>> > >>> On Mon, 1 Sept 2025, 16:04 Wei Lee, <weilee...@gmail.com> wrote: > >>> > >>>> Thanks everyone for joining the discussion! > >>>> > >>>> I think we have some consensus. These should at least apply to new > code. > >>>> > >>>> 1. Don’t use AirflowException > >>>> 2. Use Python native whenever possible > >>>> 3. Create customized exceptions instead of AirflowException > >>>> > >>>> But we’re worried about how and whether we should remove > >> AirflowException. > >>>> What I am thinking is removing `raise AirflowException`. > >>>> As for AirflowException itself, it looks somewhat challenging. > >>>> What if we do something like `CustomException(AirflowException)`, > >>>> so `except AirflowException` blocks still work, but we can gain the > >>>> benefit of making it descriptive. > >>>> > >>>> > >>>> Best, > >>>> Wei > >>>> > >>>>> On Aug 30, 2025, at 11:08 PM, Jarek Potiuk <ja...@potiuk.com> wrote: > >>>>> > >>>>>> I would say we should only worry about backcompat when there's > >> something > >>>>> to > >>>>> worry about. > >>>>> > >>>>> I think Daniel is quite right. In this case what **would** happen if > >>>>> someone relies on `try AirflowException`, the issue would be > immediate > >> to > >>>>> see and easy to fix (and we can describe it in release notes as > >>>> significant > >>>>> news. I can't easily think of a "reasonable" case where behaviour > would > >>>> be > >>>>> changed without crashing. I think that would be a blocker if we can > >> have > >>>> a > >>>>> reasonable way of using it that might change the behaviour in a way > >> that > >>>> is > >>>>> difficult to notice by the user, but when I think of potential cases > I > >>>>> cannot imagine usage that could lead to it. > >>>>> > >>>>> J > >>>>> > >>>>> > >>>>> On Wed, Aug 27, 2025 at 3:31 PM Daniel Standish > >>>>> <daniel.stand...@astronomer.io.invalid> wrote: > >>>>> > >>>>>> Generally agree with this. > >>>>>> > >>>>>> I would say we should only worry about backcompat when there's > >>>> something to > >>>>>> worry about. > >>>>>> > >>>>>> Like, if there's just a remote possibility that a user is doing > >>>> something > >>>>>> weird (e.g. in some subclass they wrote of something) then, not > really > >>>> for > >>>>>> us to worry about. Seems rather internal behavior generally. > >>>>>> > >>>>>> If we *know* for sure changing it will break airflow for certain > >>>>>> combinations that should work, that's something to address. > >>>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>> > >>>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > >
