Hi. Thank you for your comments!
I actually didn't realize that those (airflow.operators.python_operator, ...) were Airflow 1 imports. I had them in my Airflow 2 instance, and I was under the impression that DAG that is working in Airflow 2 should work in Airflow 3, but apparently not. After changing imports from (airflow.operators.python, ...), DAGs are now parsed fine. Thank you for your responses. - Eugene On Wed, Apr 2, 2025 at 4:11 AM Wei Lee <weilee...@gmail.com> wrote: > Hi Kaxil, > > > Instead handle it via ruff rules AIR2 something > > Does that mean we need to add an `AIR2` for `_operator`? In the current > implementation, they are in `AIR303` (moved to provider). > > It also raises a new thing I'd like to confirm. > We're to mark the suggested change for `airflow.operators.python` as they > are supposed to work ok on Airflow 3, but it would be better to change them > to a standard provider. > For `airflow.operators.python_operator`, it sounds like something is no > longer working in Airflow 3. If this is the case, we probably need to mark > them as required changes in Airflow 3 and suggested changes in AIR2? > > > --- > > ferruzzi, > > > But to the original topic, an AF2-AF3 ruff rule was "always" the plan > AFAIK. I've lost track of who is working on what at this point > > I'm working on the ruff thing. AIR3 things only for now, though. > > --- > > Tamara, > > > but might change to "AIR30" for crucial changes and "AIR31" for suggested > changes > > We already reached a consensus with the ruff team but will need some time > to implement it. > > https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 > > --- > > > Best, > Wei > > > On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > <tamara.finger...@astronomer.io.INVALID> wrote: > > > > Hi Eugen > > > > As others have said > > from airflow.operators.python_operator import PythonOperator > > has been deprecated a long time ago and wont work anymore in Airflow 3. > > > > from airflow.operators.python import PythonOperator > > which was the preferred import for recent years is going to get > deprecated > > for Airflow 3 but will still work in 3.0 (likely will be removed > eventually) > > > > and the new preferred import will be from standard providers: > > from airflow.providers.standard.operators.python import PythonOperator > > > > There is a ruff rule in the works ( > > https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" > but > > might change to "AIR30" for crucial changes and "AIR31" for suggested > > changes and it is still being worked on (currently it would catch the > > removed import from python_operator but recommend the deprecated > statement, > > not the one from standard providers yet). > > > > ruff check --preview --select AIR3 > > is the command right now if you want to test, but I'd wait until Airflow > 3 > > release so it is all final. There will also be upgrading guides. :) > > > > > > > > On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > <ferru...@amazon.com.invalid> > > wrote: > > > >> IMHO, if they were deprecated that long ago then adding a ruff rule is > >> just enabling the users to ignore the deprecation. They should > >> /eventually/ have to clear some of their tech debt and actually update > >> their code to modern standards, no? They've had four and half YEARS of > >> warning at this point. But to the original topic, an AF2-AF3 ruff rule > >> was "always" the plan AFAIK. I've lost track of who is working on what > at > >> this point, but I know I heard talk about that almost since the > beginning, > >> no? That seems like as good of a place as any to handle this redirect. > >> > >> > >> - ferruzzi > >> > >> > >> ________________________________ > >> From: Jarek Potiuk <ja...@potiuk.com> > >> Sent: Tuesday, April 1, 2025 9:32 AM > >> To: dev@airflow.apache.org > >> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 > >> > >> CAUTION: This email originated from outside of the organization. Do not > >> click links or open attachments unless you can confirm the sender and > know > >> the content is safe. > >> > >> > >> > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > externe. > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > pouvez > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > que > >> le contenu ne présente aucun risque. > >> > >> > >> > >> Ah.. I missed that we have those _operators ... Those indeed should have > >> been fixed a long time ago and automated conversion rules from ruff > >> should be fixing them. > >> > >> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > >> > >>> Those were deprecated 4.5+ years ago: > >>> > >>> > >> > https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py > >>> > >>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik <kaxiln...@gmail.com> wrote: > >>> > >>>> Instead handle it via ruff rules AIR2 something > >>>> > >>>> On Tue, 1 Apr 2025 at 21:44, Kaxil Naik <kaxiln...@gmail.com> wrote: > >>>> > >>>>> ` - ModuleNotFoundError: No module named > >>>>> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x > >>> old > >>>>> > >>>>> We had already stripped `_operator` from the module names in Airflow > >>>>> 2.0.0 -- so IMO there is no need to keep back-compatibility for > >>> something > >>>>> that was working 2 major versions ago > >>>>> > >>>>> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk <ja...@potiuk.com> wrote: > >>>>> > >>>>>> An example where deprection_tools are still used > >>>>>> > >>>>>> > >>> > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py > >>>>>> > >>>>>> It's rather straightforward = needs a package with __init__.py - > only > >>>>>> where > >>>>>> you list all the classes and provide redirections. It will > >>>>>> automatically raise deprecation warnings: > >>>>>> > >>>>>> from airflow.utils.deprecation_tools import add_deprecated_classes > >>>>>> > >>>>>> __deprecated_classes = { > >>>>>> "cloudwatch_task_handler": { > >>>>>> "CloudwatchTaskHandler": ( > >>>>>> > >>>>>> > >>>>>> > >>> > >> > "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler" > >>>>>> ), > >>>>>> }, > >>>>>> "es_task_handler": { > >>>>>> "ElasticsearchTaskHandler": ( > >>>>>> > >>>>>> > >>>>>> > >>> > >> > "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler" > >>>>>> ), > >>>>>> }, > >>>>>> "gcs_task_handler": { > >>>>>> "GCSTaskHandler": > >>>>>> "airflow.providers.google > >> .cloud.log.gcs_task_handler.GCSTaskHandler", > >>>>>> }, > >>>>>> "s3_task_handler": { > >>>>>> "S3TaskHandler": > >>>>>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler", > >>>>>> }, > >>>>>> "stackdriver_task_handler": { > >>>>>> "StackdriverTaskHandler": ( > >>>>>> > >>>>>> "airflow.providers.google > >>>>>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler" > >>>>>> ), > >>>>>> }, > >>>>>> "wasb_task_handler": { > >>>>>> "WasbTaskHandler": > >>>>>> > >>>>>> > >>> > >> > "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler", > >>>>>> }, > >>>>>> } > >>>>>> > >>>>>> add_deprecated_classes(__deprecated_classes, __name__) > >>>>>> > >>>>>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk <ja...@potiuk.com> > >> wrote: > >>>>>> > >>>>>>> We were going to have compatibility shims to redirect the imports - > >>>>>> with - > >>>>>>> there are few ways to do it - Ash had a little POC with module > >>> loader, > >>>>>> but > >>>>>>> I think it has some potential side effect and I think Ash abandoned > >>>>>>> the idea and I would personally prefer to use our old PEP-563 > >>> mechanism > >>>>>>> using airflow-core/src/airflow/utils/deprecation_tools.py, > >>>>>>> > >>>>>>> Very nice and small PR to implement if you want to contribute - > >> and > >>>>>> since > >>>>>>> you are testing it now with some existing DAGS it might also be a > >>> good > >>>>>> test > >>>>>>> if no redirect has been forgotten > >>>>>>> > >>>>>>> J. > >>>>>>> > >>>>>>> > >>>>>>> On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev <eu...@kosteev.com> > >>>>>> wrote: > >>>>>>> > >>>>>>>> Hi everyone. > >>>>>>>> > >>>>>>>> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and > >>> would > >>>>>>>> like > >>>>>>>> to discuss this topic. > >>>>>>>> > >>>>>>>> I took bunch of DAGs from existing Airflow 2 instances and > >> deployed > >>>>>> them > >>>>>>>> to > >>>>>>>> instance with Airflow 3 (3.0.0b4) and have bunch of import errors: > >>>>>>>> - ModuleNotFoundError: No module named > >>>>>>>> 'airflow.operators.python_operator' > >>>>>>>> - ModuleNotFoundError: No module named > >>>>>> 'airflow.operators.bash_operator' > >>>>>>>> - ImportError: cannot import name 'email_operator' from > >>>>>>>> 'airflow.operators' > >>>>>>>> - ModuleNotFoundError: No module named > >>>>>>>> 'airflow.operators.dummy_operator' > >>>>>>>> > >>>>>>>> I know that users are supposed to migrate from using > >>>>>> "airflow.operators" > >>>>>>>> to > >>>>>>>> standard/stmp/.. provider packages before upgrading to Airflow 3. > >>>>>>>> > >>>>>>>> But I also remember some discussions around keeping old imports > >>> work, > >>>>>> by > >>>>>>>> rerouting them to the correct module (similarly as we do in case > >> of > >>>>>>>> deprecated classes, etc.). > >>>>>>>> So, it will be very smooth for users to migrate to Airflow 3. > >>>>>>>> > >>>>>>>> What is our stand on this? Do we abandon "airflow.operators" usage > >>> in > >>>>>> DAGs > >>>>>>>> in Airflow 3 completely? > >>>>>>>> Or this is something that needs to be done in Airflow 3, but not > >>> yet. > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Eugene > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > > -- Eugene
