Wei, Really appreciate the update.
Thank you for the follow through, Vikram On Wed, Apr 2, 2025 at 8:37 PM Wei Lee <weilee...@gmail.com> wrote: > Yep, just want to make sure my understanding is correct. > > --- > > Follow up on the discussion of the required/suggested changes. > > I list down what we have in ruff now: > https://github.com/apache/airflow/issues/48560#issuecomment-2772148143 > (before separating them as required/suggested changes). > > And in the following 2 comments, I separate them as required changes (with > error codes AIR301, AIR302) and suggested changes (with AIR311, AIR312) > > changes in core airflow > https://github.com/apache/airflow/issues/48560#issuecomment-2772747938 > > functionality that has been moved to an airflow provider > https://github.com/apache/airflow/issues/48560#issuecomment-2772608006 > > We’re to implement the ruff changes based on this list. Please comment on > that issue if there’s anything missed or wrong. > > Best, > Wei > > > On Apr 2, 2025, at 7:51 PM, Kaxil Naik <kaxiln...@gmail.com> wrote: > > > >> Does that mean we need to add an `AIR2` for `_operator`? In the current > > implementation, they are in `AIR303` (moved to provider). > > > > Yeah it might be worth doing it, all though we should prioritize AF 2 to > AF > > 3 only for now since we have a long list of things to do. Once that is > > done, we can do AIR2x something imo > > > > On Wed, 2 Apr 2025 at 07:41, Wei Lee <weilee...@gmail.com> wrote: > > > >> Hi Kaxil, > >> > >>> Instead handle it via ruff rules AIR2 something > >> > >> Does that mean we need to add an `AIR2` for `_operator`? In the current > >> implementation, they are in `AIR303` (moved to provider). > >> > >> It also raises a new thing I'd like to confirm. > >> We're to mark the suggested change for `airflow.operators.python` as > they > >> are supposed to work ok on Airflow 3, but it would be better to change > them > >> to a standard provider. > >> For `airflow.operators.python_operator`, it sounds like something is no > >> longer working in Airflow 3. If this is the case, we probably need to > mark > >> them as required changes in Airflow 3 and suggested changes in AIR2? > >> > >> > >> --- > >> > >> ferruzzi, > >> > >>> But to the original topic, an AF2-AF3 ruff rule was "always" the plan > >> AFAIK. I've lost track of who is working on what at this point > >> > >> I'm working on the ruff thing. AIR3 things only for now, though. > >> > >> --- > >> > >> Tamara, > >> > >>> but might change to "AIR30" for crucial changes and "AIR31" for > suggested > >> changes > >> > >> We already reached a consensus with the ruff team but will need some > time > >> to implement it. > >> > >> https://github.com/astral-sh/ruff/issues/14626#issuecomment-2766146129 > >> > >> --- > >> > >> > >> Best, > >> Wei > >> > >>> On Apr 2, 2025, at 1:06 AM, Tamara Fingerlin > >> <tamara.finger...@astronomer.io.INVALID> wrote: > >>> > >>> Hi Eugen > >>> > >>> As others have said > >>> from airflow.operators.python_operator import PythonOperator > >>> has been deprecated a long time ago and wont work anymore in Airflow 3. > >>> > >>> from airflow.operators.python import PythonOperator > >>> which was the preferred import for recent years is going to get > >> deprecated > >>> for Airflow 3 but will still work in 3.0 (likely will be removed > >> eventually) > >>> > >>> and the new preferred import will be from standard providers: > >>> from airflow.providers.standard.operators.python import > PythonOperator > >>> > >>> There is a ruff rule in the works ( > >>> https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" > >> but > >>> might change to "AIR30" for crucial changes and "AIR31" for suggested > >>> changes and it is still being worked on (currently it would catch the > >>> removed import from python_operator but recommend the deprecated > >> statement, > >>> not the one from standard providers yet). > >>> > >>> ruff check --preview --select AIR3 > >>> is the command right now if you want to test, but I'd wait until > Airflow > >> 3 > >>> release so it is all final. There will also be upgrading guides. :) > >>> > >>> > >>> > >>> On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis > >> <ferru...@amazon.com.invalid> > >>> wrote: > >>> > >>>> IMHO, if they were deprecated that long ago then adding a ruff rule is > >>>> just enabling the users to ignore the deprecation. They should > >>>> /eventually/ have to clear some of their tech debt and actually update > >>>> their code to modern standards, no? They've had four and half YEARS > of > >>>> warning at this point. But to the original topic, an AF2-AF3 ruff > rule > >>>> was "always" the plan AFAIK. I've lost track of who is working on > what > >> at > >>>> this point, but I know I heard talk about that almost since the > >> beginning, > >>>> no? That seems like as good of a place as any to handle this > redirect. > >>>> > >>>> > >>>> - ferruzzi > >>>> > >>>> > >>>> ________________________________ > >>>> From: Jarek Potiuk <ja...@potiuk.com> > >>>> Sent: Tuesday, April 1, 2025 9:32 AM > >>>> To: dev@airflow.apache.org > >>>> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with > Airflow 3 > >>>> > >>>> CAUTION: This email originated from outside of the organization. Do > not > >>>> click links or open attachments unless you can confirm the sender and > >> know > >>>> the content is safe. > >>>> > >>>> > >>>> > >>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > >> externe. > >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > >> pouvez > >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain > >> que > >>>> le contenu ne présente aucun risque. > >>>> > >>>> > >>>> > >>>> Ah.. I missed that we have those _operators ... Those indeed should > have > >>>> been fixed a long time ago and automated conversion rules from ruff > >>>> should be fixing them. > >>>> > >>>> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik <kaxiln...@gmail.com> > wrote: > >>>> > >>>>> Those were deprecated 4.5+ years ago: > >>>>> > >>>>> > >>>> > >> > https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py > >>>>> > >>>>> On Tue, 1 Apr 2025 at 21:45, Kaxil Naik <kaxiln...@gmail.com> wrote: > >>>>> > >>>>>> Instead handle it via ruff rules AIR2 something > >>>>>> > >>>>>> On Tue, 1 Apr 2025 at 21:44, Kaxil Naik <kaxiln...@gmail.com> > wrote: > >>>>>> > >>>>>>> ` - ModuleNotFoundError: No module named > >>>>>>> 'airflow.operators.python_operator'` <-- those paths are Airflow > 1.x > >>>>> old > >>>>>>> > >>>>>>> We had already stripped `_operator` from the module names in > Airflow > >>>>>>> 2.0.0 -- so IMO there is no need to keep back-compatibility for > >>>>> something > >>>>>>> that was working 2 major versions ago > >>>>>>> > >>>>>>> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk <ja...@potiuk.com> > wrote: > >>>>>>> > >>>>>>>> An example where deprection_tools are still used > >>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py > >>>>>>>> > >>>>>>>> It's rather straightforward = needs a package with __init__.py - > >> only > >>>>>>>> where > >>>>>>>> you list all the classes and provide redirections. It will > >>>>>>>> automatically raise deprecation warnings: > >>>>>>>> > >>>>>>>> from airflow.utils.deprecation_tools import add_deprecated_classes > >>>>>>>> > >>>>>>>> __deprecated_classes = { > >>>>>>>> "cloudwatch_task_handler": { > >>>>>>>> "CloudwatchTaskHandler": ( > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler" > >>>>>>>> ), > >>>>>>>> }, > >>>>>>>> "es_task_handler": { > >>>>>>>> "ElasticsearchTaskHandler": ( > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler" > >>>>>>>> ), > >>>>>>>> }, > >>>>>>>> "gcs_task_handler": { > >>>>>>>> "GCSTaskHandler": > >>>>>>>> "airflow.providers.google > >>>> .cloud.log.gcs_task_handler.GCSTaskHandler", > >>>>>>>> }, > >>>>>>>> "s3_task_handler": { > >>>>>>>> "S3TaskHandler": > >>>>>>>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler", > >>>>>>>> }, > >>>>>>>> "stackdriver_task_handler": { > >>>>>>>> "StackdriverTaskHandler": ( > >>>>>>>> > >>>>>>>> "airflow.providers.google > >>>>>>>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler" > >>>>>>>> ), > >>>>>>>> }, > >>>>>>>> "wasb_task_handler": { > >>>>>>>> "WasbTaskHandler": > >>>>>>>> > >>>>>>>> > >>>>> > >>>> > >> > "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler", > >>>>>>>> }, > >>>>>>>> } > >>>>>>>> > >>>>>>>> add_deprecated_classes(__deprecated_classes, __name__) > >>>>>>>> > >>>>>>>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk <ja...@potiuk.com> > >>>> wrote: > >>>>>>>> > >>>>>>>>> We were going to have compatibility shims to redirect the > imports - > >>>>>>>> with - > >>>>>>>>> there are few ways to do it - Ash had a little POC with module > >>>>> loader, > >>>>>>>> but > >>>>>>>>> I think it has some potential side effect and I think Ash > abandoned > >>>>>>>>> the idea and I would personally prefer to use our old PEP-563 > >>>>> mechanism > >>>>>>>>> using airflow-core/src/airflow/utils/deprecation_tools.py, > >>>>>>>>> > >>>>>>>>> Very nice and small PR to implement if you want to contribute - > >>>> and > >>>>>>>> since > >>>>>>>>> you are testing it now with some existing DAGS it might also be a > >>>>> good > >>>>>>>> test > >>>>>>>>> if no redirect has been forgotten > >>>>>>>>> > >>>>>>>>> J. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev <eu...@kosteev.com> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Hi everyone. > >>>>>>>>>> > >>>>>>>>>> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and > >>>>> would > >>>>>>>>>> like > >>>>>>>>>> to discuss this topic. > >>>>>>>>>> > >>>>>>>>>> I took bunch of DAGs from existing Airflow 2 instances and > >>>> deployed > >>>>>>>> them > >>>>>>>>>> to > >>>>>>>>>> instance with Airflow 3 (3.0.0b4) and have bunch of import > errors: > >>>>>>>>>> - ModuleNotFoundError: No module named > >>>>>>>>>> 'airflow.operators.python_operator' > >>>>>>>>>> - ModuleNotFoundError: No module named > >>>>>>>> 'airflow.operators.bash_operator' > >>>>>>>>>> - ImportError: cannot import name 'email_operator' from > >>>>>>>>>> 'airflow.operators' > >>>>>>>>>> - ModuleNotFoundError: No module named > >>>>>>>>>> 'airflow.operators.dummy_operator' > >>>>>>>>>> > >>>>>>>>>> I know that users are supposed to migrate from using > >>>>>>>> "airflow.operators" > >>>>>>>>>> to > >>>>>>>>>> standard/stmp/.. provider packages before upgrading to Airflow > 3. > >>>>>>>>>> > >>>>>>>>>> But I also remember some discussions around keeping old imports > >>>>> work, > >>>>>>>> by > >>>>>>>>>> rerouting them to the correct module (similarly as we do in case > >>>> of > >>>>>>>>>> deprecated classes, etc.). > >>>>>>>>>> So, it will be very smooth for users to migrate to Airflow 3. > >>>>>>>>>> > >>>>>>>>>> What is our stand on this? Do we abandon "airflow.operators" > usage > >>>>> in > >>>>>>>> DAGs > >>>>>>>>>> in Airflow 3 completely? > >>>>>>>>>> Or this is something that needs to be done in Airflow 3, but not > >>>>> yet. > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Eugene > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> For additional commands, e-mail: dev-h...@airflow.apache.org > >> > >> > >
