Hi Eugen
As others have said
from airflow.operators.python_operator import PythonOperator
has been deprecated a long time ago and wont work anymore in Airflow 3.
from airflow.operators.python import PythonOperator
which was the preferred import for recent years is going to get deprecated
for Airflow 3 but will still work in 3.0 (likely will be removed eventually)
and the new preferred import will be from standard providers:
from airflow.providers.standard.operators.python import PythonOperator
There is a ruff rule in the works (
https://docs.astral.sh/ruff/rules/#airflow-air) it currently is "AIR3" but
might change to "AIR30" for crucial changes and "AIR31" for suggested
changes and it is still being worked on (currently it would catch the
removed import from python_operator but recommend the deprecated statement,
not the one from standard providers yet).
ruff check --preview --select AIR3
is the command right now if you want to test, but I'd wait until Airflow 3
release so it is all final. There will also be upgrading guides. :)
On Tue, Apr 1, 2025 at 6:52 PM Ferruzzi, Dennis <ferru...@amazon.com.invalid>
wrote:
> IMHO, if they were deprecated that long ago then adding a ruff rule is
> just enabling the users to ignore the deprecation. They should
> /eventually/ have to clear some of their tech debt and actually update
> their code to modern standards, no? They've had four and half YEARS of
> warning at this point. But to the original topic, an AF2-AF3 ruff rule
> was "always" the plan AFAIK. I've lost track of who is working on what at
> this point, but I know I heard talk about that almost since the beginning,
> no? That seems like as good of a place as any to handle this redirect.
>
>
> - ferruzzi
>
>
> ________________________________
> From: Jarek Potiuk <ja...@potiuk.com>
> Sent: Tuesday, April 1, 2025 9:32 AM
> To: dev@airflow.apache.org
> Subject: RE: [EXT] [DISCUSSION] Airflow 2 DAGs compatible with Airflow 3
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur externe.
> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne pouvez
> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas certain que
> le contenu ne présente aucun risque.
>
>
>
> Ah.. I missed that we have those _operators ... Those indeed should have
> been fixed a long time ago and automated conversion rules from ruff
> should be fixing them.
>
> On Tue, Apr 1, 2025 at 6:17 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
>
> > Those were deprecated 4.5+ years ago:
> >
> >
> https://github.com/apache/airflow/blob/2.0.0/airflow/operators/python_operator.py
> >
> > On Tue, 1 Apr 2025 at 21:45, Kaxil Naik <kaxiln...@gmail.com> wrote:
> >
> > > Instead handle it via ruff rules AIR2 something
> > >
> > > On Tue, 1 Apr 2025 at 21:44, Kaxil Naik <kaxiln...@gmail.com> wrote:
> > >
> > >> ` - ModuleNotFoundError: No module named
> > >> 'airflow.operators.python_operator'` <-- those paths are Airflow 1.x
> > old
> > >>
> > >> We had already stripped `_operator` from the module names in Airflow
> > >> 2.0.0 -- so IMO there is no need to keep back-compatibility for
> > something
> > >> that was working 2 major versions ago
> > >>
> > >> On Tue, 1 Apr 2025 at 17:23, Jarek Potiuk <ja...@potiuk.com> wrote:
> > >>
> > >>> An example where deprection_tools are still used
> > >>>
> > >>>
> >
> https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/log/__init__.py
> > >>>
> > >>> It's rather straightforward = needs a package with __init__.py - only
> > >>> where
> > >>> you list all the classes and provide redirections. It will
> > >>> automatically raise deprecation warnings:
> > >>>
> > >>> from airflow.utils.deprecation_tools import add_deprecated_classes
> > >>>
> > >>> __deprecated_classes = {
> > >>> "cloudwatch_task_handler": {
> > >>> "CloudwatchTaskHandler": (
> > >>>
> > >>>
> > >>>
> >
> "airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler"
> > >>> ),
> > >>> },
> > >>> "es_task_handler": {
> > >>> "ElasticsearchTaskHandler": (
> > >>>
> > >>>
> > >>>
> >
> "airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler"
> > >>> ),
> > >>> },
> > >>> "gcs_task_handler": {
> > >>> "GCSTaskHandler":
> > >>> "airflow.providers.google
> .cloud.log.gcs_task_handler.GCSTaskHandler",
> > >>> },
> > >>> "s3_task_handler": {
> > >>> "S3TaskHandler":
> > >>> "airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler",
> > >>> },
> > >>> "stackdriver_task_handler": {
> > >>> "StackdriverTaskHandler": (
> > >>>
> > >>> "airflow.providers.google
> > >>> .cloud.log.stackdriver_task_handler.StackdriverTaskHandler"
> > >>> ),
> > >>> },
> > >>> "wasb_task_handler": {
> > >>> "WasbTaskHandler":
> > >>>
> > >>>
> >
> "airflow.providers.microsoft.azure.log.wasb_task_handler.WasbTaskHandler",
> > >>> },
> > >>> }
> > >>>
> > >>> add_deprecated_classes(__deprecated_classes, __name__)
> > >>>
> > >>> On Tue, Apr 1, 2025 at 1:49 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > >>>
> > >>> > We were going to have compatibility shims to redirect the imports -
> > >>> with -
> > >>> > there are few ways to do it - Ash had a little POC with module
> > loader,
> > >>> but
> > >>> > I think it has some potential side effect and I think Ash abandoned
> > >>> > the idea and I would personally prefer to use our old PEP-563
> > mechanism
> > >>> > using airflow-core/src/airflow/utils/deprecation_tools.py,
> > >>> >
> > >>> > Very nice and small PR to implement if you want to contribute -
> and
> > >>> since
> > >>> > you are testing it now with some existing DAGS it might also be a
> > good
> > >>> test
> > >>> > if no redirect has been forgotten
> > >>> >
> > >>> > J.
> > >>> >
> > >>> >
> > >>> > On Tue, Apr 1, 2025 at 1:32 PM Eugen Kosteev <eu...@kosteev.com>
> > >>> wrote:
> > >>> >
> > >>> >> Hi everyone.
> > >>> >>
> > >>> >> I am testing compatibility of Airflow 2 DAGs with Airflow 3, and
> > would
> > >>> >> like
> > >>> >> to discuss this topic.
> > >>> >>
> > >>> >> I took bunch of DAGs from existing Airflow 2 instances and
> deployed
> > >>> them
> > >>> >> to
> > >>> >> instance with Airflow 3 (3.0.0b4) and have bunch of import errors:
> > >>> >> - ModuleNotFoundError: No module named
> > >>> >> 'airflow.operators.python_operator'
> > >>> >> - ModuleNotFoundError: No module named
> > >>> 'airflow.operators.bash_operator'
> > >>> >> - ImportError: cannot import name 'email_operator' from
> > >>> >> 'airflow.operators'
> > >>> >> - ModuleNotFoundError: No module named
> > >>> >> 'airflow.operators.dummy_operator'
> > >>> >>
> > >>> >> I know that users are supposed to migrate from using
> > >>> "airflow.operators"
> > >>> >> to
> > >>> >> standard/stmp/.. provider packages before upgrading to Airflow 3.
> > >>> >>
> > >>> >> But I also remember some discussions around keeping old imports
> > work,
> > >>> by
> > >>> >> rerouting them to the correct module (similarly as we do in case
> of
> > >>> >> deprecated classes, etc.).
> > >>> >> So, it will be very smooth for users to migrate to Airflow 3.
> > >>> >>
> > >>> >> What is our stand on this? Do we abandon "airflow.operators" usage
> > in
> > >>> DAGs
> > >>> >> in Airflow 3 completely?
> > >>> >> Or this is something that needs to be done in Airflow 3, but not
> > yet.
> > >>> >>
> > >>> >> --
> > >>> >> Eugene
> > >>> >>
> > >>> >
> > >>>
> > >>
> >
>
