Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Wei, Really appreciate the update. Thank you for the follow through, Vikram On Wed, Apr 2, 2025 at 8:37 PM Wei Lee wrote: > Yep, just want to make sure my understanding is correct. > > --- > > Follow up on the discussion of the required/suggested changes. > > I list down what we have in ruff now: > https://github.com/apache/airflow/issues/48560#issuecomment-2772148143 > (before separating them as required/suggested changes). > > And in the following 2 comments, I separate them as required changes (with > error codes AIR301, AIR302) and suggested changes (with AIR311, AIR312) > > changes in core airflow > https://github.com/apache/airflow/issues/48560#issuecomment-2772747938 > > functionality that has been moved to an airflow provider > https://github.com/apache/airflow/issues/48560#issuecomment-2772608006 > > We’re to implement the ruff changes based on this list. Please comment on > that issue if there’s anything missed or wrong. > > Best, > Wei > > > On Apr 2, 2025, at 7:51 PM, Kaxil Naik wrote: > > > >> Does that mean we need to add an `AIR2` for `_operator`? In the current > > implementation, they are in `AIR303` (moved to provider). > > > > Yeah it might be worth doing it, all though we should prioritize AF 2 to > AF > > 3 only for now since we have a long list of things to do. Once that is > > done, we can do AIR2x something imo > > > > On Wed, 2 Apr 2025 at 07:41, Wei Lee wrote: > > > >> Hi Kaxil, > >> > >>> Instead handle it via ruff rules AIR2 something > >> > >> Does that mean we need to add an `AIR2` for `_operator`? In the current > >> implementation, they are in `AIR303` (moved to provider). > >> > >> It also raises a new thing I'd like to confirm. > >> We're to mark the suggested change for `airflow.operators.python` as > they > >> are supposed to work ok on Airflow 3, but it would be better to change > them > >> to a standard provider. > >> For `airflow.operators.python_operator`, it sounds like something is no > >> longer working in Airflow 3. If this is the case, we probably need to > mark > >> them as required changes in Airflow 3 and suggested changes in AIR2? > >> > >> > >> --- > >> > >> ferruzzi, > >> > >>> But to the original topic, an AF2-AF3 ruff rule was "always" the plan > >> AFAIK. I've lost track of who is working on what at this point > >> > >> I'm working on the ruff thing. AIR3 things only for now, though. > >> > >> --- > >> > >> Tamara, > >> > >>> but might change to "AIR30" for crucial changes and "AIR31" for > suggested > >> changes > >> > >> We already reached a consensus with the ruff team but will need some > time > >> to implement it. > >> > >> https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 > >> > >> --- > >> > >> > >> Best, > >> Wei > >> > >>> On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > >> wrote: > >>> > >>> Hi Eugen > >>> > >>> As others have said > >>> from airflow.operators.python_operator import PythonOperator > >>> has been deprecated a long time ago and wont work anymore in Airflow 3. > >>> > >>> from airflow.operators.python import PythonOperator > >>> which was the preferred import for recent years is going to get > >> deprecated > >>> for Airflow 3 but will still work in 3.0 (likely will be removed > >> eventually) > >>> > >>> and the new preferred import will be from standard providers: > >>> from airflow.providers.standard.operators.python import > PythonOperator > >>> > >>> There is a ruff rule in the works ( > >>> https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" > >> but > >>> might change to "AIR30" for crucial changes and "AIR31" for suggested > >>> changes and it is still being worked on (currently it would catch the > >>> removed import from python_operator but recommend the deprecated > >> statement, > >>> not the one from standard providers yet). > >>> > >>> ruff check --preview --select AIR3 > >>> is the command right now if you want to test, but I'd wait until > Airflow > >> 3 > >>> release so it is all final. There will also be upgrading guides. :) > >>> > >>> > >>> > >>> On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > >> > >>> wrote: > >>> > IMHO, if they were deprecated that long ago then adding a ruff rule is > just enabling the users to ignore the deprecation. They should > /eventually/ have to clear some of their tech debt and actually update > their code to modern standards, no? They've had four and half YEARS > of > warning at this point. But to the original topic, an AF2-AF3 ruff > rule > was "always" the plan AFAIK. I've lost track of who is working on > what > >> at > this point, but I know I heard talk about that almost since the > >> beginning, > no? That seems like as good of a place as any to handle this > redirect. > > > - ferruzzi > > > > From: Jarek Potiuk > Sent: Tuesday, April 1, 2025 9:32 AM > To: dev@airflow.apache.org > Subject
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Yep, just want to make sure my understanding is correct. --- Follow up on the discussion of the required/suggested changes. I list down what we have in ruff now: https://github.com/apache/airflow/issues/48560#issuecomment-2772148143 (before separating them as required/suggested changes). And in the following 2 comments, I separate them as required changes (with error codes AIR301, AIR302) and suggested changes (with AIR311, AIR312) changes in core airflow https://github.com/apache/airflow/issues/48560#issuecomment-2772747938 functionality that has been moved to an airflow provider https://github.com/apache/airflow/issues/48560#issuecomment-2772608006 We’re to implement the ruff changes based on this list. Please comment on that issue if there’s anything missed or wrong. Best, Wei > On Apr 2, 2025, at 7:51 PM, Kaxil Naik wrote: > >> Does that mean we need to add an `AIR2` for `_operator`? In the current > implementation, they are in `AIR303` (moved to provider). > > Yeah it might be worth doing it, all though we should prioritize AF 2 to AF > 3 only for now since we have a long list of things to do. Once that is > done, we can do AIR2x something imo > > On Wed, 2 Apr 2025 at 07:41, Wei Lee wrote: > >> Hi Kaxil, >> >>> Instead handle it via ruff rules AIR2 something >> >> Does that mean we need to add an `AIR2` for `_operator`? In the current >> implementation, they are in `AIR303` (moved to provider). >> >> It also raises a new thing I'd like to confirm. >> We're to mark the suggested change for `airflow.operators.python` as they >> are supposed to work ok on Airflow 3, but it would be better to change them >> to a standard provider. >> For `airflow.operators.python_operator`, it sounds like something is no >> longer working in Airflow 3. If this is the case, we probably need to mark >> them as required changes in Airflow 3 and suggested changes in AIR2? >> >> >> --- >> >> ferruzzi, >> >>> But to the original topic, an AF2-AF3 ruff rule was "always" the plan >> AFAIK. I've lost track of who is working on what at this point >> >> I'm working on the ruff thing. AIR3 things only for now, though. >> >> --- >> >> Tamara, >> >>> but might change to "AIR30" for crucial changes and "AIR31" for suggested >> changes >> >> We already reached a consensus with the ruff team but will need some time >> to implement it. >> >> https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 >> >> --- >> >> >> Best, >> Wei >> >>> On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin >> wrote: >>> >>> Hi Eugen >>> >>> As others have said >>> from airflow.operators.python_operator import PythonOperator >>> has been deprecated a long time ago and wont work anymore in Airflow 3. >>> >>> from airflow.operators.python import PythonOperator >>> which was the preferred import for recent years is going to get >> deprecated >>> for Airflow 3 but will still work in 3.0 (likely will be removed >> eventually) >>> >>> and the new preferred import will be from standard providers: >>> from airflow.providers.standard.operators.python import PythonOperator >>> >>> There is a ruff rule in the works ( >>> https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" >> but >>> might change to "AIR30" for crucial changes and "AIR31" for suggested >>> changes and it is still being worked on (currently it would catch the >>> removed import from python_operator but recommend the deprecated >> statement, >>> not the one from standard providers yet). >>> >>> ruff check --preview --select AIR3 >>> is the command right now if you want to test, but I'd wait until Airflow >> 3 >>> release so it is all final. There will also be upgrading guides. :) >>> >>> >>> >>> On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis >> >>> wrote: >>> IMHO, if they were deprecated that long ago then adding a ruff rule is just enabling the users to ignore the deprecation. They should /eventually/ have to clear some of their tech debt and actually update their code to modern standards, no? They've had four and half YEARS of warning at this point. But to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. I've lost track of who is working on what >> at this point, but I know I heard talk about that almost since the >> beginning, no? That seems like as good of a place as any to handle this redirect. - ferruzzi From: Jarek Potiuk Sent: Tuesday, April 1, 2025 9:32 AM To: dev@airflow.apache.org Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and >> know the content is safe. AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. Ne cliquez sur
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
>Does that mean we need to add an `AIR2` for `_operator`? In the current implementation, they are in `AIR303` (moved to provider). Yeah it might be worth doing it, all though we should prioritize AF 2 to AF 3 only for now since we have a long list of things to do. Once that is done, we can do AIR2x something imo On Wed, 2 Apr 2025 at 07:41, Wei Lee wrote: > Hi Kaxil, > > > Instead handle it via ruff rules AIR2 something > > Does that mean we need to add an `AIR2` for `_operator`? In the current > implementation, they are in `AIR303` (moved to provider). > > It also raises a new thing I'd like to confirm. > We're to mark the suggested change for `airflow.operators.python` as they > are supposed to work ok on Airflow 3, but it would be better to change them > to a standard provider. > For `airflow.operators.python_operator`, it sounds like something is no > longer working in Airflow 3. If this is the case, we probably need to mark > them as required changes in Airflow 3 and suggested changes in AIR2? > > > --- > > ferruzzi, > > > But to the original topic, an AF2-AF3 ruff rule was "always" the plan > AFAIK. I've lost track of who is working on what at this point > > I'm working on the ruff thing. AIR3 things only for now, though. > > --- > > Tamara, > > > but might change to "AIR30" for crucial changes and "AIR31" for suggested > changes > > We already reached a consensus with the ruff team but will need some time > to implement it. > > https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 > > --- > > > Best, > Wei > > > On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > wrote: > > > > Hi Eugen > > > > As others have said > >from airflow.operators.python_operator import PythonOperator > > has been deprecated a long time ago and wont work anymore in Airflow 3. > > > >from airflow.operators.python import PythonOperator > > which was the preferred import for recent years is going to get > deprecated > > for Airflow 3 but will still work in 3.0 (likely will be removed > eventually) > > > > and the new preferred import will be from standard providers: > >from airflow.providers.standard.operators.python import PythonOperator > > > > There is a ruff rule in the works ( > > https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" > but > > might change to "AIR30" for crucial changes and "AIR31" for suggested > > changes and it is still being worked on (currently it would catch the > > removed import from python_operator but recommend the deprecated > statement, > > not the one from standard providers yet). > > > >ruff check --preview --select AIR3 > > is the command right now if you want to test, but I'd wait until Airflow > 3 > > release so it is all final. There will also be upgrading guides. :) > > > > > > > > On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > > > wrote: > > > >> IMHO, if they were deprecated that long ago then adding a ruff rule is > >> just enabling the users to ignore the deprecation. They should > >> /eventually/ have to clear some of their tech debt and actually update > >> their code to modern standards, no? They've had four and half YEARS of > >> warning at this point. But to the original topic, an AF2-AF3 ruff rule > >> was "always" the plan AFAIK. I've lost track of who is working on what > at > >> this point, but I know I heard talk about that almost since the > beginning, > >> no? That seems like as good of a place as any to handle this redirect. > >> > >> > >> - ferruzzi > >> > >> > >> > >> From: Jarek Potiuk > >> Sent: Tuesday, April 1, 2025 9:32 AM > >> To: dev@airflow.apache.org > >> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 > >> > >> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >> > >> > >> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > >> le contenu ne présente aucun risque. > >> > >> > >> > >> Ah.. I missed that we have those _operators ... Those indeed should have > >> been fixed a long time ago and automated conversion rules from ruff > >> should be fixing them. > >> > >> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote: > >> > >>> Those were deprecated 4.5+ years ago: > >>> > >>> > >> > https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py > >>> > >>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote: > >>> > Instead handle it via ruff rules AIR2 something > > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote: > > > ` - ModuleNotFoundError: No module named > > 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x > >>> old > > > > We had already stripp
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Hi. Thank you for your comments! I actually didn't realize that those (airflow.operators.python_operator, ...) were Airflow 1 imports. I had them in my Airflow 2 instance, and I was under the impression that DAG that is working in Airflow 2 should work in Airflow 3, but apparently not. After changing imports from (airflow.operators.python, ...), DAGs are now parsed fine. Thank you for your responses. - Eugene On Wed, Apr 2, 2025 at 4:11 AM Wei Lee wrote: > Hi Kaxil, > > > Instead handle it via ruff rules AIR2 something > > Does that mean we need to add an `AIR2` for `_operator`? In the current > implementation, they are in `AIR303` (moved to provider). > > It also raises a new thing I'd like to confirm. > We're to mark the suggested change for `airflow.operators.python` as they > are supposed to work ok on Airflow 3, but it would be better to change them > to a standard provider. > For `airflow.operators.python_operator`, it sounds like something is no > longer working in Airflow 3. If this is the case, we probably need to mark > them as required changes in Airflow 3 and suggested changes in AIR2? > > > --- > > ferruzzi, > > > But to the original topic, an AF2-AF3 ruff rule was "always" the plan > AFAIK. I've lost track of who is working on what at this point > > I'm working on the ruff thing. AIR3 things only for now, though. > > --- > > Tamara, > > > but might change to "AIR30" for crucial changes and "AIR31" for suggested > changes > > We already reached a consensus with the ruff team but will need some time > to implement it. > > https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 > > --- > > > Best, > Wei > > > On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > wrote: > > > > Hi Eugen > > > > As others have said > >from airflow.operators.python_operator import PythonOperator > > has been deprecated a long time ago and wont work anymore in Airflow 3. > > > >from airflow.operators.python import PythonOperator > > which was the preferred import for recent years is going to get > deprecated > > for Airflow 3 but will still work in 3.0 (likely will be removed > eventually) > > > > and the new preferred import will be from standard providers: > >from airflow.providers.standard.operators.python import PythonOperator > > > > There is a ruff rule in the works ( > > https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" > but > > might change to "AIR30" for crucial changes and "AIR31" for suggested > > changes and it is still being worked on (currently it would catch the > > removed import from python_operator but recommend the deprecated > statement, > > not the one from standard providers yet). > > > >ruff check --preview --select AIR3 > > is the command right now if you want to test, but I'd wait until Airflow > 3 > > release so it is all final. There will also be upgrading guides. :) > > > > > > > > On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > > > wrote: > > > >> IMHO, if they were deprecated that long ago then adding a ruff rule is > >> just enabling the users to ignore the deprecation. They should > >> /eventually/ have to clear some of their tech debt and actually update > >> their code to modern standards, no? They've had four and half YEARS of > >> warning at this point. But to the original topic, an AF2-AF3 ruff rule > >> was "always" the plan AFAIK. I've lost track of who is working on what > at > >> this point, but I know I heard talk about that almost since the > beginning, > >> no? That seems like as good of a place as any to handle this redirect. > >> > >> > >> - ferruzzi > >> > >> > >> > >> From: Jarek Potiuk > >> Sent: Tuesday, April 1, 2025 9:32 AM > >> To: dev@airflow.apache.org > >> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 > >> > >> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >> > >> > >> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > >> le contenu ne présente aucun risque. > >> > >> > >> > >> Ah.. I missed that we have those _operators ... Those indeed should have > >> been fixed a long time ago and automated conversion rules from ruff > >> should be fixing them. > >> > >> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote: > >> > >>> Those were deprecated 4.5+ years ago: > >>> > >>> > >> > https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py > >>> > >>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote: > >>> > Instead handle it via ruff rules AIR2 something > > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote: > > > ` - ModuleNotFoundError: No module named > > 'airflow.ope
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Hi Kaxil, > Instead handle it via ruff rules AIR2 something Does that mean we need to add an `AIR2` for `_operator`? In the current implementation, they are in `AIR303` (moved to provider). It also raises a new thing I'd like to confirm. We're to mark the suggested change for `airflow.operators.python` as they are supposed to work ok on Airflow 3, but it would be better to change them to a standard provider. For `airflow.operators.python_operator`, it sounds like something is no longer working in Airflow 3. If this is the case, we probably need to mark them as required changes in Airflow 3 and suggested changes in AIR2? --- ferruzzi, > But to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. > I've lost track of who is working on what at this point I'm working on the ruff thing. AIR3 things only for now, though. --- Tamara, > but might change to "AIR30" for crucial changes and "AIR31" for suggested changes We already reached a consensus with the ruff team but will need some time to implement it. https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 --- Best, Wei > On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > wrote: > > Hi Eugen > > As others have said >from airflow.operators.python_operator import PythonOperator > has been deprecated a long time ago and wont work anymore in Airflow 3. > >from airflow.operators.python import PythonOperator > which was the preferred import for recent years is going to get deprecated > for Airflow 3 but will still work in 3.0 (likely will be removed eventually) > > and the new preferred import will be from standard providers: >from airflow.providers.standard.operators.python import PythonOperator > > There is a ruff rule in the works ( > https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" but > might change to "AIR30" for crucial changes and "AIR31" for suggested > changes and it is still being worked on (currently it would catch the > removed import from python_operator but recommend the deprecated statement, > not the one from standard providers yet). > >ruff check --preview --select AIR3 > is the command right now if you want to test, but I'd wait until Airflow 3 > release so it is all final. There will also be upgrading guides. :) > > > > On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > wrote: > >> IMHO, if they were deprecated that long ago then adding a ruff rule is >> just enabling the users to ignore the deprecation. They should >> /eventually/ have to clear some of their tech debt and actually update >> their code to modern standards, no? They've had four and half YEARS of >> warning at this point. But to the original topic, an AF2-AF3 ruff rule >> was "always" the plan AFAIK. I've lost track of who is working on what at >> this point, but I know I heard talk about that almost since the beginning, >> no? That seems like as good of a place as any to handle this redirect. >> >> >> - ferruzzi >> >> >> >> From: Jarek Potiuk >> Sent: Tuesday, April 1, 2025 9:32 AM >> To: dev@airflow.apache.org >> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que >> le contenu ne présente aucun risque. >> >> >> >> Ah.. I missed that we have those _operators ... Those indeed should have >> been fixed a long time ago and automated conversion rules from ruff >> should be fixing them. >> >> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote: >> >>> Those were deprecated 4.5+ years ago: >>> >>> >> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py >>> >>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote: >>> Instead handle it via ruff rules AIR2 something On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote: > ` - ModuleNotFoundError: No module named > 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x >>> old > > We had already stripped `_operator` from the module names in Airflow > 2.0.0 -- so IMO there is no need to keep back-compatibility for >>> something > that was working 2 major versions ago > > On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote: > >> An example where deprection_tools are still used >> >> >>> >> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py >> >> It's rather straightforward = needs a package with __init__.py - only >> where >> you list all the classes and provide redirections. It will >
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Hi Eugen
As others have said
from airflow.operators.python_operator import PythonOperator
has been deprecated a long time ago and wont work anymore in Airflow 3.
from airflow.operators.python import PythonOperator
which was the preferred import for recent years is going to get deprecated
for Airflow 3 but will still work in 3.0 (likely will be removed eventually)
and the new preferred import will be from standard providers:
from airflow.providers.standard.operators.python import PythonOperator
There is a ruff rule in the works (
https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" but
might change to "AIR30" for crucial changes and "AIR31" for suggested
changes and it is still being worked on (currently it would catch the
removed import from python_operator but recommend the deprecated statement,
not the one from standard providers yet).
ruff check --preview --select AIR3
is the command right now if you want to test, but I'd wait until Airflow 3
release so it is all final. There will also be upgrading guides. :)
On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis
wrote:
> IMHO, if they were deprecated that long ago then adding a ruff rule is
> just enabling the users to ignore the deprecation. They should
> /eventually/ have to clear some of their tech debt and actually update
> their code to modern standards, no? They've had four and half YEARS of
> warning at this point. But to the original topic, an AF2-AF3 ruff rule
> was "always" the plan AFAIK. I've lost track of who is working on what at
> this point, but I know I heard talk about that almost since the beginning,
> no? That seems like as good of a place as any to handle this redirect.
>
>
> - ferruzzi
>
>
>
> From: Jarek Potiuk
> Sent: Tuesday, April 1, 2025 9:32 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> Ah.. I missed that we have those _operators ... Those indeed should have
> been fixed a long time ago and automated conversion rules from ruff
> should be fixing them.
>
> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote:
>
> > Those were deprecated 4.5+ years ago:
> >
> >
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
> >
> > On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
> >
> > > Instead handle it via ruff rules AIR2 something
> > >
> > > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> > >
> > >> ` - ModuleNotFoundError: No module named
> > >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> > old
> > >>
> > >> We had already stripped `_operator` from the module names in Airflow
> > >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> > something
> > >> that was working 2 major versions ago
> > >>
> > >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> > >>
> > >>> An example where deprection_tools are still used
> > >>>
> > >>>
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> > >>>
> > >>> It's rather straightforward = needs a package with __init__.py - only
> > >>> where
> > >>> you list all the classes and provide redirections. It will
> > >>> automatically raise deprecation warnings:
> > >>>
> > >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> > >>>
> > >>> __deprecated_classes = {
> > >>> "cloudwatch_task_handler": {
> > >>> "CloudwatchTaskHandler": (
> > >>>
> > >>>
> > >>>
> >
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> > >>> ),
> > >>> },
> > >>> "es_task_handler": {
> > >>> "ElasticsearchTaskHandler": (
> > >>>
> > >>>
> > >>>
> >
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> > >>> ),
> > >>> },
> > >>> "gcs_task_handler": {
> > >>> "GCSTaskHandler":
> > >>> "airflow.providers.google
> .cloud.log.gcs_task_handler.GCSTaskHandler",
> > >>> },
> > >>> "s3_task_handler": {
> > >>> "S3TaskHandler":
> > >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> > >>> },
> > >>> "stackdriver_task_handler": {
> > >>> "StackdriverTaskHandler": (
> > >>>
> > >>> "airflow.providers.google
> > >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> > >>> ),
> > >>> },
> > >>> "wasb_task_handler": {
> > >>> "WasbTaskHandler":
> > >>>
> > >>>
> >
> "airflow.providers.microsoft.azure.log.was
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
IMHO, if they were deprecated that long ago then adding a ruff rule is just
enabling the users to ignore the deprecation. They should /eventually/ have to
clear some of their tech debt and actually update their code to modern
standards, no? They've had four and half YEARS of warning at this point. But
to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. I've
lost track of who is working on what at this point, but I know I heard talk
about that almost since the beginning, no? That seems like as good of a place
as any to handle this redirect.
- ferruzzi
From: Jarek Potiuk
Sent: Tuesday, April 1, 2025 9:32 AM
To: dev@airflow.apache.org
Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. Ne
cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez pas
confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que le
contenu ne présente aucun risque.
Ah.. I missed that we have those _operators ... Those indeed should have
been fixed a long time ago and automated conversion rules from ruff
should be fixing them.
On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote:
> Those were deprecated 4.5+ years ago:
>
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
>
> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
>
> > Instead handle it via ruff rules AIR2 something
> >
> > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> >
> >> ` - ModuleNotFoundError: No module named
> >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> old
> >>
> >> We had already stripped `_operator` from the module names in Airflow
> >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> something
> >> that was working 2 major versions ago
> >>
> >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> >>
> >>> An example where deprection_tools are still used
> >>>
> >>>
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> >>>
> >>> It's rather straightforward = needs a package with __init__.py - only
> >>> where
> >>> you list all the classes and provide redirections. It will
> >>> automatically raise deprecation warnings:
> >>>
> >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> >>>
> >>> __deprecated_classes = {
> >>> "cloudwatch_task_handler": {
> >>> "CloudwatchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> >>> ),
> >>> },
> >>> "es_task_handler": {
> >>> "ElasticsearchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> >>> ),
> >>> },
> >>> "gcs_task_handler": {
> >>> "GCSTaskHandler":
> >>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
> >>> },
> >>> "s3_task_handler": {
> >>> "S3TaskHandler":
> >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> >>> },
> >>> "stackdriver_task_handler": {
> >>> "StackdriverTaskHandler": (
> >>>
> >>> "airflow.providers.google
> >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> >>> ),
> >>> },
> >>> "wasb_task_handler": {
> >>> "WasbTaskHandler":
> >>>
> >>>
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> >>> },
> >>> }
> >>>
> >>> add_deprecated_classes(__deprecated_classes, __name__)
> >>>
> >>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
> >>>
> >>> > We were going to have compatibility shims to redirect the imports -
> >>> with -
> >>> > there are few ways to do it - Ash had a little POC with module
> loader,
> >>> but
> >>> > I think it has some potential side effect and I think Ash abandoned
> >>> > the idea and I would personally prefer to use our old PEP-563
> mechanism
> >>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> >>> >
> >>> > Very nice and small PR to implement if you want to contribute - and
> >>> since
> >>> > you are testing it now with some existing DAGS it might also be a
> good
> >>> test
> >>> > if no redirect has been forgotten
> >>> >
> >>> > J.
> >>> >
> >>> >
> >>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev
> >>> wrote:
> >>> >
> >>> >> Hi everyone.
> >>> >>
> >>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and
> would
> >>> >> like
> >>> >> to discuss this topic.
> >>> >>
> >>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
> >>> them
> >>> >> to
> >>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> >>> >> - ModuleNotFoundError: No module named
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Ah.. I missed that we have those _operators ... Those indeed should have
been fixed a long time ago and automated conversion rules from ruff
should be fixing them.
On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik wrote:
> Those were deprecated 4.5+ years ago:
>
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
>
> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
>
> > Instead handle it via ruff rules AIR2 something
> >
> > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> >
> >> ` - ModuleNotFoundError: No module named
> >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> old
> >>
> >> We had already stripped `_operator` from the module names in Airflow
> >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> something
> >> that was working 2 major versions ago
> >>
> >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> >>
> >>> An example where deprection_tools are still used
> >>>
> >>>
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> >>>
> >>> It's rather straightforward = needs a package with __init__.py - only
> >>> where
> >>> you list all the classes and provide redirections. It will
> >>> automatically raise deprecation warnings:
> >>>
> >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> >>>
> >>> __deprecated_classes = {
> >>> "cloudwatch_task_handler": {
> >>> "CloudwatchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> >>> ),
> >>> },
> >>> "es_task_handler": {
> >>> "ElasticsearchTaskHandler": (
> >>>
> >>>
> >>>
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> >>> ),
> >>> },
> >>> "gcs_task_handler": {
> >>> "GCSTaskHandler":
> >>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
> >>> },
> >>> "s3_task_handler": {
> >>> "S3TaskHandler":
> >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> >>> },
> >>> "stackdriver_task_handler": {
> >>> "StackdriverTaskHandler": (
> >>>
> >>> "airflow.providers.google
> >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> >>> ),
> >>> },
> >>> "wasb_task_handler": {
> >>> "WasbTaskHandler":
> >>>
> >>>
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> >>> },
> >>> }
> >>>
> >>> add_deprecated_classes(__deprecated_classes, __name__)
> >>>
> >>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
> >>>
> >>> > We were going to have compatibility shims to redirect the imports -
> >>> with -
> >>> > there are few ways to do it - Ash had a little POC with module
> loader,
> >>> but
> >>> > I think it has some potential side effect and I think Ash abandoned
> >>> > the idea and I would personally prefer to use our old PEP-563
> mechanism
> >>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> >>> >
> >>> > Very nice and small PR to implement if you want to contribute - and
> >>> since
> >>> > you are testing it now with some existing DAGS it might also be a
> good
> >>> test
> >>> > if no redirect has been forgotten
> >>> >
> >>> > J.
> >>> >
> >>> >
> >>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev
> >>> wrote:
> >>> >
> >>> >> Hi everyone.
> >>> >>
> >>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and
> would
> >>> >> like
> >>> >> to discuss this topic.
> >>> >>
> >>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
> >>> them
> >>> >> to
> >>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> >>> >> - ModuleNotFoundError: No module named
> >>> >> 'airflow.operators.python_operator'
> >>> >> - ModuleNotFoundError: No module named
> >>> 'airflow.operators.bash_operator'
> >>> >> - ImportError: cannot import name 'email_operator' from
> >>> >> 'airflow.operators'
> >>> >> - ModuleNotFoundError: No module named
> >>> >> 'airflow.operators.dummy_operator'
> >>> >>
> >>> >> I know that users are supposed to migrate from using
> >>> "airflow.operators"
> >>> >> to
> >>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
> >>> >>
> >>> >> But I also remember some discussions around keeping old imports
> work,
> >>> by
> >>> >> rerouting them to the correct module (similarly as we do in case of
> >>> >> deprecated classes, etc.).
> >>> >> So, it will be very smooth for users to migrate to Airflow 3.
> >>> >>
> >>> >> What is our stand on this? Do we abandon "airflow.operators" usage
> in
> >>> DAGs
> >>> >> in Airflow 3 completely?
> >>> >> Or this is something that needs to be done in Airflow 3, but not
> yet.
> >>> >>
> >>> >> --
> >>> >> Eugene
> >>> >>
> >>> >
> >>>
> >>
>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Those were deprecated 4.5+ years ago:
https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
On Tue, 1 Apr 2025 at 21:45, Kaxil Naik wrote:
> Instead handle it via ruff rules AIR2 something
>
> On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
>
>> ` - ModuleNotFoundError: No module named
>> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x old
>>
>> We had already stripped `_operator` from the module names in Airflow
>> 2.0.0 -- so IMO there is no need to keep back-compatibility for something
>> that was working 2 major versions ago
>>
>> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
>>
>>> An example where deprection_tools are still used
>>>
>>> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
>>>
>>> It's rather straightforward = needs a package with __init__.py - only
>>> where
>>> you list all the classes and provide redirections. It will
>>> automatically raise deprecation warnings:
>>>
>>> from airflow.utils.deprecation_tools import add_deprecated_classes
>>>
>>> __deprecated_classes = {
>>> "cloudwatch_task_handler": {
>>> "CloudwatchTaskHandler": (
>>>
>>>
>>> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
>>> ),
>>> },
>>> "es_task_handler": {
>>> "ElasticsearchTaskHandler": (
>>>
>>>
>>> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
>>> ),
>>> },
>>> "gcs_task_handler": {
>>> "GCSTaskHandler":
>>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
>>> },
>>> "s3_task_handler": {
>>> "S3TaskHandler":
>>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
>>> },
>>> "stackdriver_task_handler": {
>>> "StackdriverTaskHandler": (
>>>
>>> "airflow.providers.google
>>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
>>> ),
>>> },
>>> "wasb_task_handler": {
>>> "WasbTaskHandler":
>>>
>>> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
>>> },
>>> }
>>>
>>> add_deprecated_classes(__deprecated_classes, __name__)
>>>
>>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
>>>
>>> > We were going to have compatibility shims to redirect the imports -
>>> with -
>>> > there are few ways to do it - Ash had a little POC with module loader,
>>> but
>>> > I think it has some potential side effect and I think Ash abandoned
>>> > the idea and I would personally prefer to use our old PEP-563 mechanism
>>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
>>> >
>>> > Very nice and small PR to implement if you want to contribute - and
>>> since
>>> > you are testing it now with some existing DAGS it might also be a good
>>> test
>>> > if no redirect has been forgotten
>>> >
>>> > J.
>>> >
>>> >
>>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev
>>> wrote:
>>> >
>>> >> Hi everyone.
>>> >>
>>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
>>> >> like
>>> >> to discuss this topic.
>>> >>
>>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
>>> them
>>> >> to
>>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
>>> >> - ModuleNotFoundError: No module named
>>> >> 'airflow.operators.python_operator'
>>> >> - ModuleNotFoundError: No module named
>>> 'airflow.operators.bash_operator'
>>> >> - ImportError: cannot import name 'email_operator' from
>>> >> 'airflow.operators'
>>> >> - ModuleNotFoundError: No module named
>>> >> 'airflow.operators.dummy_operator'
>>> >>
>>> >> I know that users are supposed to migrate from using
>>> "airflow.operators"
>>> >> to
>>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
>>> >>
>>> >> But I also remember some discussions around keeping old imports work,
>>> by
>>> >> rerouting them to the correct module (similarly as we do in case of
>>> >> deprecated classes, etc.).
>>> >> So, it will be very smooth for users to migrate to Airflow 3.
>>> >>
>>> >> What is our stand on this? Do we abandon "airflow.operators" usage in
>>> DAGs
>>> >> in Airflow 3 completely?
>>> >> Or this is something that needs to be done in Airflow 3, but not yet.
>>> >>
>>> >> --
>>> >> Eugene
>>> >>
>>> >
>>>
>>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
Instead handle it via ruff rules AIR2 something
On Tue, 1 Apr 2025 at 21:44, Kaxil Naik wrote:
> ` - ModuleNotFoundError: No module named
> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x old
>
> We had already stripped `_operator` from the module names in Airflow 2.0.0
> -- so IMO there is no need to keep back-compatibility for something that
> was working 2 major versions ago
>
> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
>
>> An example where deprection_tools are still used
>>
>> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
>>
>> It's rather straightforward = needs a package with __init__.py - only
>> where
>> you list all the classes and provide redirections. It will
>> automatically raise deprecation warnings:
>>
>> from airflow.utils.deprecation_tools import add_deprecated_classes
>>
>> __deprecated_classes = {
>> "cloudwatch_task_handler": {
>> "CloudwatchTaskHandler": (
>>
>>
>> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
>> ),
>> },
>> "es_task_handler": {
>> "ElasticsearchTaskHandler": (
>>
>>
>> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
>> ),
>> },
>> "gcs_task_handler": {
>> "GCSTaskHandler":
>> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
>> },
>> "s3_task_handler": {
>> "S3TaskHandler":
>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
>> },
>> "stackdriver_task_handler": {
>> "StackdriverTaskHandler": (
>>
>> "airflow.providers.google
>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
>> ),
>> },
>> "wasb_task_handler": {
>> "WasbTaskHandler":
>> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
>> },
>> }
>>
>> add_deprecated_classes(__deprecated_classes, __name__)
>>
>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
>>
>> > We were going to have compatibility shims to redirect the imports -
>> with -
>> > there are few ways to do it - Ash had a little POC with module loader,
>> but
>> > I think it has some potential side effect and I think Ash abandoned
>> > the idea and I would personally prefer to use our old PEP-563 mechanism
>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
>> >
>> > Very nice and small PR to implement if you want to contribute - and
>> since
>> > you are testing it now with some existing DAGS it might also be a good
>> test
>> > if no redirect has been forgotten
>> >
>> > J.
>> >
>> >
>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote:
>> >
>> >> Hi everyone.
>> >>
>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
>> >> like
>> >> to discuss this topic.
>> >>
>> >> I took bunch of DAGs from existing Airflow 2 instances and deployed
>> them
>> >> to
>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
>> >> - ModuleNotFoundError: No module named
>> >> 'airflow.operators.python_operator'
>> >> - ModuleNotFoundError: No module named
>> 'airflow.operators.bash_operator'
>> >> - ImportError: cannot import name 'email_operator' from
>> >> 'airflow.operators'
>> >> - ModuleNotFoundError: No module named
>> >> 'airflow.operators.dummy_operator'
>> >>
>> >> I know that users are supposed to migrate from using
>> "airflow.operators"
>> >> to
>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
>> >>
>> >> But I also remember some discussions around keeping old imports work,
>> by
>> >> rerouting them to the correct module (similarly as we do in case of
>> >> deprecated classes, etc.).
>> >> So, it will be very smooth for users to migrate to Airflow 3.
>> >>
>> >> What is our stand on this? Do we abandon "airflow.operators" usage in
>> DAGs
>> >> in Airflow 3 completely?
>> >> Or this is something that needs to be done in Airflow 3, but not yet.
>> >>
>> >> --
>> >> Eugene
>> >>
>> >
>>
>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
` - ModuleNotFoundError: No module named
'airflow.operators.python_operator'` <-- those paths are Airflow 1.x old
We had already stripped `_operator` from the module names in Airflow 2.0.0
-- so IMO there is no need to keep back-compatibility for something that
was working 2 major versions ago
On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk wrote:
> An example where deprection_tools are still used
>
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
>
> It's rather straightforward = needs a package with __init__.py - only where
> you list all the classes and provide redirections. It will
> automatically raise deprecation warnings:
>
> from airflow.utils.deprecation_tools import add_deprecated_classes
>
> __deprecated_classes = {
> "cloudwatch_task_handler": {
> "CloudwatchTaskHandler": (
>
>
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> ),
> },
> "es_task_handler": {
> "ElasticsearchTaskHandler": (
>
>
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> ),
> },
> "gcs_task_handler": {
> "GCSTaskHandler":
> "airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
> },
> "s3_task_handler": {
> "S3TaskHandler":
> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> },
> "stackdriver_task_handler": {
> "StackdriverTaskHandler": (
>
> "airflow.providers.google
> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> ),
> },
> "wasb_task_handler": {
> "WasbTaskHandler":
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> },
> }
>
> add_deprecated_classes(__deprecated_classes, __name__)
>
> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
>
> > We were going to have compatibility shims to redirect the imports - with
> -
> > there are few ways to do it - Ash had a little POC with module loader,
> but
> > I think it has some potential side effect and I think Ash abandoned
> > the idea and I would personally prefer to use our old PEP-563 mechanism
> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> >
> > Very nice and small PR to implement if you want to contribute - and
> since
> > you are testing it now with some existing DAGS it might also be a good
> test
> > if no redirect has been forgotten
> >
> > J.
> >
> >
> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote:
> >
> >> Hi everyone.
> >>
> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
> >> like
> >> to discuss this topic.
> >>
> >> I took bunch of DAGs from existing Airflow 2 instances and deployed them
> >> to
> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> >> - ModuleNotFoundError: No module named
> >> 'airflow.operators.python_operator'
> >> - ModuleNotFoundError: No module named
> 'airflow.operators.bash_operator'
> >> - ImportError: cannot import name 'email_operator' from
> >> 'airflow.operators'
> >> - ModuleNotFoundError: No module named
> >> 'airflow.operators.dummy_operator'
> >>
> >> I know that users are supposed to migrate from using "airflow.operators"
> >> to
> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
> >>
> >> But I also remember some discussions around keeping old imports work, by
> >> rerouting them to the correct module (similarly as we do in case of
> >> deprecated classes, etc.).
> >> So, it will be very smooth for users to migrate to Airflow 3.
> >>
> >> What is our stand on this? Do we abandon "airflow.operators" usage in
> DAGs
> >> in Airflow 3 completely?
> >> Or this is something that needs to be done in Airflow 3, but not yet.
> >>
> >> --
> >> Eugene
> >>
> >
>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
An example where deprection_tools are still used
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
It's rather straightforward = needs a package with __init__.py - only where
you list all the classes and provide redirections. It will
automatically raise deprecation warnings:
from airflow.utils.deprecation_tools import add_deprecated_classes
__deprecated_classes = {
"cloudwatch_task_handler": {
"CloudwatchTaskHandler": (
"airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
),
},
"es_task_handler": {
"ElasticsearchTaskHandler": (
"airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
),
},
"gcs_task_handler": {
"GCSTaskHandler":
"airflow.providers.google.cloud.log.gcs_task_handler.GCSTaskHandler",
},
"s3_task_handler": {
"S3TaskHandler":
"airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
},
"stackdriver_task_handler": {
"StackdriverTaskHandler": (
"airflow.providers.google.cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
),
},
"wasb_task_handler": {
"WasbTaskHandler":
"airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
},
}
add_deprecated_classes(__deprecated_classes, __name__)
On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk wrote:
> We were going to have compatibility shims to redirect the imports - with -
> there are few ways to do it - Ash had a little POC with module loader, but
> I think it has some potential side effect and I think Ash abandoned
> the idea and I would personally prefer to use our old PEP-563 mechanism
> using airflow-core/src/airflow/utils/deprecation_tools.py,
>
> Very nice and small PR to implement if you want to contribute - and since
> you are testing it now with some existing DAGS it might also be a good test
> if no redirect has been forgotten
>
> J.
>
>
> On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote:
>
>> Hi everyone.
>>
>> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would
>> like
>> to discuss this topic.
>>
>> I took bunch of DAGs from existing Airflow 2 instances and deployed them
>> to
>> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
>> - ModuleNotFoundError: No module named
>> 'airflow.operators.python_operator'
>> - ModuleNotFoundError: No module named 'airflow.operators.bash_operator'
>> - ImportError: cannot import name 'email_operator' from
>> 'airflow.operators'
>> - ModuleNotFoundError: No module named
>> 'airflow.operators.dummy_operator'
>>
>> I know that users are supposed to migrate from using "airflow.operators"
>> to
>> standard/stmp/.. provider packages before upgrading to Airflow 3.
>>
>> But I also remember some discussions around keeping old imports work, by
>> rerouting them to the correct module (similarly as we do in case of
>> deprecated classes, etc.).
>> So, it will be very smooth for users to migrate to Airflow 3.
>>
>> What is our stand on this? Do we abandon "airflow.operators" usage in DAGs
>> in Airflow 3 completely?
>> Or this is something that needs to be done in Airflow 3, but not yet.
>>
>> --
>> Eugene
>>
>
Re: [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
We were going to have compatibility shims to redirect the imports - with - there are few ways to do it - Ash had a little POC with module loader, but I think it has some potential side effect and I think Ash abandoned the idea and I would personally prefer to use our old PEP-563 mechanism using airflow-core/src/airflow/utils/deprecation_tools.py, Very nice and small PR to implement if you want to contribute - and since you are testing it now with some existing DAGS it might also be a good test if no redirect has been forgotten J. On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev wrote: > Hi everyone. > > I am testing compatibility of Airflow 2 DAGs with Airflow 3, and would like > to discuss this topic. > > I took bunch of DAGs from existing Airflow 2 instances and deployed them to > instance with Airflow 3 (3.0.0b4) and have bunch of import errors: > - ModuleNotFoundError: No module named > 'airflow.operators.python_operator' > - ModuleNotFoundError: No module named 'airflow.operators.bash_operator' > - ImportError: cannot import name 'email_operator' from > 'airflow.operators' > - ModuleNotFoundError: No module named 'airflow.operators.dummy_operator' > > I know that users are supposed to migrate from using "airflow.operators" to > standard/stmp/.. provider packages before upgrading to Airflow 3. > > But I also remember some discussions around keeping old imports work, by > rerouting them to the correct module (similarly as we do in case of > deprecated classes, etc.). > So, it will be very smooth for users to migrate to Airflow 3. > > What is our stand on this? Do we abandon "airflow.operators" usage in DAGs > in Airflow 3 completely? > Or this is something that needs to be done in Airflow 3, but not yet. > > -- > Eugene >
