Yep, just want to make sure my understanding is correct. ---
Follow up on the discussion of the required/suggested changes. I list down what we have in ruff now: https://github.com/apache/airflow/issues/48560#issuecomment-2772148143 (before separating them as required/suggested changes). And in the following 2 comments, I separate them as required changes (with error codes AIR301, AIR302) and suggested changes (with AIR311, AIR312) changes in core airflow https://github.com/apache/airflow/issues/48560#issuecomment-2772747938 functionality that has been moved to an airflow provider https://github.com/apache/airflow/issues/48560#issuecomment-2772608006 We’re to implement the ruff changes based on this list. Please comment on that issue if there’s anything missed or wrong. Best, Wei > On Apr 2, 2025, at 7:51 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > >> Does that mean we need to add an `AIR2` for `_operator`? In the current > implementation, they are in `AIR303` (moved to provider). > > Yeah it might be worth doing it, all though we should prioritize AF 2 to AF > 3 only for now since we have a long list of things to do. Once that is > done, we can do AIR2x something imo > > On Wed, 2 Apr 2025 at 07:41, Wei Lee <weilee...@gmail.com> wrote: > >> Hi Kaxil, >> >>> Instead handle it via ruff rules AIR2 something >> >> Does that mean we need to add an `AIR2` for `_operator`? In the current >> implementation, they are in `AIR303` (moved to provider). >> >> It also raises a new thing I'd like to confirm. >> We're to mark the suggested change for `airflow.operators.python` as they >> are supposed to work ok on Airflow 3, but it would be better to change them >> to a standard provider. >> For `airflow.operators.python_operator`, it sounds like something is no >> longer working in Airflow 3. If this is the case, we probably need to mark >> them as required changes in Airflow 3 and suggested changes in AIR2? >> >> >> --- >> >> ferruzzi, >> >>> But to the original topic, an AF2-AF3 ruff rule was "always" the plan >> AFAIK. I've lost track of who is working on what at this point >> >> I'm working on the ruff thing. AIR3 things only for now, though. >> >> --- >> >> Tamara, >> >>> but might change to "AIR30" for crucial changes and "AIR31" for suggested >> changes >> >> We already reached a consensus with the ruff team but will need some time >> to implement it. >> >> https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 >> >> --- >> >> >> Best, >> Wei >> >>> On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin >> <tamara.finger...@astronomer.io.INVALID> wrote: >>> >>> Hi Eugen >>> >>> As others have said >>> from airflow.operators.python_operator import PythonOperator >>> has been deprecated a long time ago and wont work anymore in Airflow 3. >>> >>> from airflow.operators.python import PythonOperator >>> which was the preferred import for recent years is going to get >> deprecated >>> for Airflow 3 but will still work in 3.0 (likely will be removed >> eventually) >>> >>> and the new preferred import will be from standard providers: >>> from airflow.providers.standard.operators.python import PythonOperator >>> >>> There is a ruff rule in the works ( >>> https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" >> but >>> might change to "AIR30" for crucial changes and "AIR31" for suggested >>> changes and it is still being worked on (currently it would catch the >>> removed import from python_operator but recommend the deprecated >> statement, >>> not the one from standard providers yet). >>> >>> ruff check --preview --select AIR3 >>> is the command right now if you want to test, but I'd wait until Airflow >> 3 >>> release so it is all final. There will also be upgrading guides. :) >>> >>> >>> >>> On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis >> <ferru...@amazon.com.invalid> >>> wrote: >>> >>>> IMHO, if they were deprecated that long ago then adding a ruff rule is >>>> just enabling the users to ignore the deprecation. They should >>>> /eventually/ have to clear some of their tech debt and actually update >>>> their code to modern standards, no? They've had four and half YEARS of >>>> warning at this point. But to the original topic, an AF2-AF3 ruff rule >>>> was "always" the plan AFAIK. I've lost track of who is working on what >> at >>>> this point, but I know I heard talk about that almost since the >> beginning, >>>> no? That seems like as good of a place as any to handle this redirect. >>>> >>>> >>>> - ferruzzi >>>> >>>> >>>> ________________________________ >>>> From: Jarek Potiuk <ja...@potiuk.com> >>>> Sent: Tuesday, April 1, 2025 9:32 AM >>>> To: dev@airflow.apache.org >>>> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3 >>>> >>>> CAUTION: This email originated from outside of the organization. Do not >>>> click links or open attachments unless you can confirm the sender and >> know >>>> the content is safe. >>>> >>>> >>>> >>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur >> externe. >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne >> pouvez >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain >> que >>>> le contenu ne présente aucun risque. >>>> >>>> >>>> >>>> Ah.. I missed that we have those _operators ... Those indeed should have >>>> been fixed a long time ago and automated conversion rules from ruff >>>> should be fixing them. >>>> >>>> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik <kaxiln...@gmail.com> wrote: >>>> >>>>> Those were deprecated 4.5+ years ago: >>>>> >>>>> >>>> >> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py >>>>> >>>>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik <kaxiln...@gmail.com> wrote: >>>>> >>>>>> Instead handle it via ruff rules AIR2 something >>>>>> >>>>>> On Tue, 1 Apr 2025 at 21:44, Kaxil Naik <kaxiln...@gmail.com> wrote: >>>>>> >>>>>>> ` - ModuleNotFoundError: No module named >>>>>>> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x >>>>> old >>>>>>> >>>>>>> We had already stripped `_operator` from the module names in Airflow >>>>>>> 2.0.0 -- so IMO there is no need to keep back-compatibility for >>>>> something >>>>>>> that was working 2 major versions ago >>>>>>> >>>>>>> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk <ja...@potiuk.com> wrote: >>>>>>> >>>>>>>> An example where deprection_tools are still used >>>>>>>> >>>>>>>> >>>>> >>>> >> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py >>>>>>>> >>>>>>>> It's rather straightforward = needs a package with __init__.py - >> only >>>>>>>> where >>>>>>>> you list all the classes and provide redirections. It will >>>>>>>> automatically raise deprecation warnings: >>>>>>>> >>>>>>>> from airflow.utils.deprecation_tools import add_deprecated_classes >>>>>>>> >>>>>>>> __deprecated_classes = { >>>>>>>> "cloudwatch_task_handler": { >>>>>>>> "CloudwatchTaskHandler": ( >>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>> >> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler" >>>>>>>> ), >>>>>>>> }, >>>>>>>> "es_task_handler": { >>>>>>>> "ElasticsearchTaskHandler": ( >>>>>>>> >>>>>>>> >>>>>>>> >>>>> >>>> >> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler" >>>>>>>> ), >>>>>>>> }, >>>>>>>> "gcs_task_handler": { >>>>>>>> "GCSTaskHandler": >>>>>>>> "airflow.providers.google >>>> .cloud.log.gcs_task_handler.GCSTaskHandler", >>>>>>>> }, >>>>>>>> "s3_task_handler": { >>>>>>>> "S3TaskHandler": >>>>>>>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler", >>>>>>>> }, >>>>>>>> "stackdriver_task_handler": { >>>>>>>> "StackdriverTaskHandler": ( >>>>>>>> >>>>>>>> "airflow.providers.google >>>>>>>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler" >>>>>>>> ), >>>>>>>> }, >>>>>>>> "wasb_task_handler": { >>>>>>>> "WasbTaskHandler": >>>>>>>> >>>>>>>> >>>>> >>>> >> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler", >>>>>>>> }, >>>>>>>> } >>>>>>>> >>>>>>>> add_deprecated_classes(__deprecated_classes, __name__) >>>>>>>> >>>>>>>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk <ja...@potiuk.com> >>>> wrote: >>>>>>>> >>>>>>>>> We were going to have compatibility shims to redirect the imports - >>>>>>>> with - >>>>>>>>> there are few ways to do it - Ash had a little POC with module >>>>> loader, >>>>>>>> but >>>>>>>>> I think it has some potential side effect and I think Ash abandoned >>>>>>>>> the idea and I would personally prefer to use our old PEP-563 >>>>> mechanism >>>>>>>>> using airflow-core/src/airflow/utils/deprecation_tools.py, >>>>>>>>> >>>>>>>>> Very nice and small PR to implement if you want to contribute - >>>> and >>>>>>>> since >>>>>>>>> you are testing it now with some existing DAGS it might also be a >>>>> good >>>>>>>> test >>>>>>>>> if no redirect has been forgotten >>>>>>>>> >>>>>>>>> J. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev <eu...@kosteev.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi everyone. >>>>>>>>>> >>>>>>>>>> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and >>>>> would >>>>>>>>>> like >>>>>>>>>> to discuss this topic. >>>>>>>>>> >>>>>>>>>> I took bunch of DAGs from existing Airflow 2 instances and >>>> deployed >>>>>>>> them >>>>>>>>>> to >>>>>>>>>> instance with Airflow 3 (3.0.0b4) and have bunch of import errors: >>>>>>>>>> - ModuleNotFoundError: No module named >>>>>>>>>> 'airflow.operators.python_operator' >>>>>>>>>> - ModuleNotFoundError: No module named >>>>>>>> 'airflow.operators.bash_operator' >>>>>>>>>> - ImportError: cannot import name 'email_operator' from >>>>>>>>>> 'airflow.operators' >>>>>>>>>> - ModuleNotFoundError: No module named >>>>>>>>>> 'airflow.operators.dummy_operator' >>>>>>>>>> >>>>>>>>>> I know that users are supposed to migrate from using >>>>>>>> "airflow.operators" >>>>>>>>>> to >>>>>>>>>> standard/stmp/.. provider packages before upgrading to Airflow 3. >>>>>>>>>> >>>>>>>>>> But I also remember some discussions around keeping old imports >>>>> work, >>>>>>>> by >>>>>>>>>> rerouting them to the correct module (similarly as we do in case >>>> of >>>>>>>>>> deprecated classes, etc.). >>>>>>>>>> So, it will be very smooth for users to migrate to Airflow 3. >>>>>>>>>> >>>>>>>>>> What is our stand on this? Do we abandon "airflow.operators" usage >>>>> in >>>>>>>> DAGs >>>>>>>>>> in Airflow 3 completely? >>>>>>>>>> Or this is something that needs to be done in Airflow 3, but not >>>>> yet. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Eugene >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>
