Hi Kaxil, > Instead handle it via ruff rules AIR2 something
Does that mean we need to add an `AIR2` for `_operator`? In the current implementation, they are in `AIR303` (moved to provider). It also raises a new thing I'd like to confirm. We're to mark the suggested change for `airflow.operators.python` as they are supposed to work ok on Airflow 3, but it would be better to change them to a standard provider. For `airflow.operators.python_operator`, it sounds like something is no longer working in Airflow 3. If this is the case, we probably need to mark them as required changes in Airflow 3 and suggested changes in AIR2? --- ferruzzi, > But to the original topic, an AF2-AF3 ruff rule was "always" the plan AFAIK. > I've lost track of who is working on what at this point I'm working on the ruff thing. AIR3 things only for now, though. --- Tamara, > but might change to "AIR30" for crucial changes and "AIR31" for suggested changes We already reached a consensus with the ruff team but will need some time to implement it. https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 --- Best, Wei > On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > <tamara.finger...@astronomer.io.INVALID> wrote: > > Hi Eugen > > As others have said > from airflow.operators.python_operator import PythonOperator > has been deprecated a long time ago and wont work anymore in Airflow 3. > > from airflow.operators.python import PythonOperator > which was the preferred import for recent years is going to get deprecated > for Airflow 3 but will still work in 3.0 (likely will be removed eventually) > > and the new preferred import will be from standard providers: > from airflow.providers.standard.operators.python import PythonOperator > > There is a ruff rule in the works ( > https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" but > might change to "AIR30" for crucial changes and "AIR31" for suggested > changes and it is still being worked on (currently it would catch the > removed import from python_operator but recommend the deprecated statement, > not the one from standard providers yet). > > ruff check --preview --select AIR3 > is the command right now if you want to test, but I'd wait until Airflow 3 > release so it is all final. There will also be upgrading guides. :) > > > > On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis <ferru...@amazon.com.invalid> > wrote: > >> IMHO, if they were deprecated that long ago then adding a ruff rule is >> just enabling the users to ignore the deprecation. They should >> /eventually/ have to clear some of their tech debt and actually update >> their code to modern standards, no? They've had four and half YEARS of >> warning at this point. But to the original topic, an AF2-AF3 ruff rule >> was "always" the plan AFAIK. I've lost track of who is working on what at >> this point, but I know I heard talk about that almost since the beginning, >> no? That seems like as good of a place as any to handle this redirect. >> >> >> - ferruzzi >> >> >> ________________________________ >> From: Jarek Potiuk <ja...@potiuk.com> >> Sent: Tuesday, April 1, 2025 9:32 AM >> To: dev@airflow.apache.org >> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 >> >> CAUTION: This email originated from outside of the organization. Do not >> click links or open attachments unless you can confirm the sender and know >> the content is safe. >> >> >> >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe. >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que >> le contenu ne présente aucun risque. >> >> >> >> Ah.. I missed that we have those _operators ... Those indeed should have >> been fixed a long time ago and automated conversion rules from ruff >> should be fixing them. >> >> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >> >>> Those were deprecated 4.5+ years ago: >>> >>> >> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py >>> >>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik <kaxiln...@gmail.com> wrote: >>> >>>> Instead handle it via ruff rules AIR2 something >>>> >>>> On Tue, 1 Apr 2025 at 21:44, Kaxil Naik <kaxiln...@gmail.com> wrote: >>>> >>>>> ` - ModuleNotFoundError: No module named >>>>> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x >>> old >>>>> >>>>> We had already stripped `_operator` from the module names in Airflow >>>>> 2.0.0 -- so IMO there is no need to keep back-compatibility for >>> something >>>>> that was working 2 major versions ago >>>>> >>>>> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk <ja...@potiuk.com> wrote: >>>>> >>>>>> An example where deprection_tools are still used >>>>>> >>>>>> >>> >> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py >>>>>> >>>>>> It's rather straightforward = needs a package with __init__.py - only >>>>>> where >>>>>> you list all the classes and provide redirections. It will >>>>>> automatically raise deprecation warnings: >>>>>> >>>>>> from airflow.utils.deprecation_tools import add_deprecated_classes >>>>>> >>>>>> __deprecated_classes = { >>>>>> "cloudwatch_task_handler": { >>>>>> "CloudwatchTaskHandler": ( >>>>>> >>>>>> >>>>>> >>> >> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler" >>>>>> ), >>>>>> }, >>>>>> "es_task_handler": { >>>>>> "ElasticsearchTaskHandler": ( >>>>>> >>>>>> >>>>>> >>> >> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler" >>>>>> ), >>>>>> }, >>>>>> "gcs_task_handler": { >>>>>> "GCSTaskHandler": >>>>>> "airflow.providers.google >> .cloud.log.gcs_task_handler.GCSTaskHandler", >>>>>> }, >>>>>> "s3_task_handler": { >>>>>> "S3TaskHandler": >>>>>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler", >>>>>> }, >>>>>> "stackdriver_task_handler": { >>>>>> "StackdriverTaskHandler": ( >>>>>> >>>>>> "airflow.providers.google >>>>>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler" >>>>>> ), >>>>>> }, >>>>>> "wasb_task_handler": { >>>>>> "WasbTaskHandler": >>>>>> >>>>>> >>> >> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler", >>>>>> }, >>>>>> } >>>>>> >>>>>> add_deprecated_classes(__deprecated_classes, __name__) >>>>>> >>>>>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk <ja...@potiuk.com> >> wrote: >>>>>> >>>>>>> We were going to have compatibility shims to redirect the imports - >>>>>> with - >>>>>>> there are few ways to do it - Ash had a little POC with module >>> loader, >>>>>> but >>>>>>> I think it has some potential side effect and I think Ash abandoned >>>>>>> the idea and I would personally prefer to use our old PEP-563 >>> mechanism >>>>>>> using airflow-core/src/airflow/utils/deprecation_tools.py, >>>>>>> >>>>>>> Very nice and small PR to implement if you want to contribute - >> and >>>>>> since >>>>>>> you are testing it now with some existing DAGS it might also be a >>> good >>>>>> test >>>>>>> if no redirect has been forgotten >>>>>>> >>>>>>> J. >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev <eu...@kosteev.com> >>>>>> wrote: >>>>>>> >>>>>>>> Hi everyone. >>>>>>>> >>>>>>>> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and >>> would >>>>>>>> like >>>>>>>> to discuss this topic. >>>>>>>> >>>>>>>> I took bunch of DAGs from existing Airflow 2 instances and >> deployed >>>>>> them >>>>>>>> to >>>>>>>> instance with Airflow 3 (3.0.0b4) and have bunch of import errors: >>>>>>>> - ModuleNotFoundError: No module named >>>>>>>> 'airflow.operators.python_operator' >>>>>>>> - ModuleNotFoundError: No module named >>>>>> 'airflow.operators.bash_operator' >>>>>>>> - ImportError: cannot import name 'email_operator' from >>>>>>>> 'airflow.operators' >>>>>>>> - ModuleNotFoundError: No module named >>>>>>>> 'airflow.operators.dummy_operator' >>>>>>>> >>>>>>>> I know that users are supposed to migrate from using >>>>>> "airflow.operators" >>>>>>>> to >>>>>>>> standard/stmp/.. provider packages before upgrading to Airflow 3. >>>>>>>> >>>>>>>> But I also remember some discussions around keeping old imports >>> work, >>>>>> by >>>>>>>> rerouting them to the correct module (similarly as we do in case >> of >>>>>>>> deprecated classes, etc.). >>>>>>>> So, it will be very smooth for users to migrate to Airflow 3. >>>>>>>> >>>>>>>> What is our stand on this? Do we abandon "airflow.operators" usage >>> in >>>>>> DAGs >>>>>>>> in Airflow 3 completely? >>>>>>>> Or this is something that needs to be done in Airflow 3, but not >>> yet. >>>>>>>> >>>>>>>> -- >>>>>>>> Eugene >>>>>>>> >>>>>>> >>>>>> >>>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
