[
https://issues.apache.org/jira/browse/AIRFLOW-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jarek Potiuk updated AIRFLOW-6003:
----------------------------------
Description:
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI:
An example of such pylint problem:
[https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification&utm_source=github_status]
{code:java}
************* Module airflow.utils.log.json_formatter
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
{code}
The problem is that airflow's __init__ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor.__init__.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor) import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number
of files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module and a
number of classes (such as AirflowException) should be moved out from __init__
package and imported from where they belong (so for example AirflowException
should be universally imported from airflow.exceptions).
was:
There are cyclic imports detected seemingly randomly by pylint checks when some
of the PRs are run in CI:
An example of such pylint problem:
[https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification&utm_source=github_status]
************* Module airflow.utils.log.json_formatter
1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_generator) (cyclic-import)
1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.pod_launcher) (cyclic-import)
1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
(airflow.executors -> airflow.executors.kubernetes_executor ->
airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
(cyclic-import)
The problem is that airflow's __init__ contains a few convenience imports
(AirflowException, Executors etc.) but it also imports a number of packages
(for example kubernetes_executor) that in turn import the airflow package
objects - for example airflow.Executor. This leads to cyclic imports if you
import first the executors before airflow. Similar problem happens with
executor.__init__.py containing class "Executors" imported by all executors but
at the same time some of the executors (for example KubernetesExecutor) import
the very same Executor class.
This might happen in pylint checks in pre-commit because they split a number of
files they process between the multiple threads you have at your machine and
sometimes it might happen that the files are imported in different order.
As a solution, the executors "list" should be moved to a separate module and a
number of classes (such as AirflowException) should be moved out from __init__
package and imported from where they belong (so for example AirflowException
should be universally imported from airflow.exceptions).
> Cyclic imports detected by pylint randomly
> ------------------------------------------
>
> Key: AIRFLOW-6003
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6003
> Project: Apache Airflow
> Issue Type: Bug
> Components: ci
> Affects Versions: 2.0.0
> Reporter: Jarek Potiuk
> Priority: Major
>
> There are cyclic imports detected seemingly randomly by pylint checks when
> some of the PRs are run in CI:
> An example of such pylint problem:
> [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification&utm_source=github_status]
>
>
> {code:java}
> ************* Module airflow.utils.log.json_formatter
> airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
> (airflow.executors -> airflow.executors.kubernetes_executor ->
> airflow.kubernetes.pod_generator) (cyclic-import)
> airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow ->
> airflow.executors -> airflow.executors.kubernetes_executor ->
> airflow.kubernetes.pod_launcher) (cyclic-import)
> airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import
> (airflow.executors -> airflow.executors.kubernetes_executor ->
> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator)
> (cyclic-import)
>
> {code}
>
> The problem is that airflow's __init__ contains a few convenience imports
> (AirflowException, Executors etc.) but it also imports a number of packages
> (for example kubernetes_executor) that in turn import the airflow package
> objects - for example airflow.Executor. This leads to cyclic imports if you
> import first the executors before airflow. Similar problem happens with
> executor.__init__.py containing class "Executors" imported by all executors
> but at the same time some of the executors (for example KubernetesExecutor)
> import the very same Executor class.
>
> This might happen in pylint checks in pre-commit because they split a number
> of files they process between the multiple threads you have at your machine
> and sometimes it might happen that the files are imported in different order.
>
> As a solution, the executors "list" should be moved to a separate module and
> a number of classes (such as AirflowException) should be moved out from
> __init__ package and imported from where they belong (so for example
> AirflowException should be universally imported from airflow.exceptions).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
