[jira] [Assigned] (AIRFLOW-5704) Docker scripts/image for kind kubernetes tests can be optimised
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5704: - Assignee: Jarek Potiuk > Docker scripts/image for kind kubernetes tests can be optimised > --- > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be optimised: > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker scripts/image for kind kubernetes tests can be optimised
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Description: The docker CI image for kind tests can be improved * Kubernetes Version and all the installation of docker + kubectl + kind can be moved to ci_install* script * Running kubernetes scripts should be possible from within breeze without special environment * --env breeze switch should be removed * ENV variable should disappear * "bare" environment should be replaced by --no-deps switch was: The docker CI image for kind tests can be optimised: * Kubernetes Version and all the installation of docker + kubectl + kind can be moved to ci_install* script * Running kubernetes scripts should be possible from within breeze without special environment * --env breeze switch should be removed * ENV variable should disappear * "bare" environment should be replaced by --no-deps switch > Docker scripts/image for kind kubernetes tests can be optimised > --- > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be improved > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker scripts for kind kubernetes tests can be improved
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Description: The docker CI image for kind tests can be improved * Kubernetes Version and all the installation of docker + kubectl + kind can be added back * Running kubernetes scripts should be possible from within breeze without special "kubernetes" environment * --env breeze switch should be removed * "bare" environment should be replaced by --no-deps switch * ENV variable should disappear was: The docker CI image for kind tests can be improved * Kubernetes Version and all the installation of docker + kubectl + kind can be moved to ci_install* script * Running kubernetes scripts should be possible from within breeze without special environment * --env breeze switch should be removed * ENV variable should disappear * "bare" environment should be replaced by --no-deps switch > Docker scripts for kind kubernetes tests can be improved > > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be improved > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be added back > * Running kubernetes scripts should be possible from within breeze without > special "kubernetes" environment > * --env breeze switch should be removed > * "bare" environment should be replaced by --no-deps switch > * ENV variable should disappear > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker scripts for kind kubernetes tests can be improved
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Summary: Docker scripts for kind kubernetes tests can be improved (was: Docker scripts/image for kind kubernetes tests can be optimised) > Docker scripts for kind kubernetes tests can be improved > > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be improved > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5839) Upgrade pre-commit to use latest version of pre-commit and hook
Jarek Potiuk created AIRFLOW-5839: - Summary: Upgrade pre-commit to use latest version of pre-commit and hook Key: AIRFLOW-5839 URL: https://issues.apache.org/jira/browse/AIRFLOW-5839 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 1.10.6, 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5830) Get rid of slim image
[ https://issues.apache.org/jira/browse/AIRFLOW-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5830: - Assignee: Jarek Potiuk > Get rid of slim image > - > > Key: AIRFLOW-5830 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5830 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > We can get rid of the slim image and use the ci image everywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5839) Upgrade pre-commit to use latest version of pre-commit and hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5839: - Assignee: Jarek Potiuk > Upgrade pre-commit to use latest version of pre-commit and hook > --- > > Key: AIRFLOW-5839 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5839 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5829) Get rid of checklicence image
[ https://issues.apache.org/jira/browse/AIRFLOW-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5829: - Assignee: Jarek Potiuk > Get rid of checklicence image > - > > Key: AIRFLOW-5829 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5829 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.5 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > We can get rid of checklicence image and run checklicence checks using the > standard CI image -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5839) Upgrade pre-commit to use latest version of pre-commit and hook
[ https://issues.apache.org/jira/browse/AIRFLOW-5839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5839. --- Fix Version/s: 1.10.7 Resolution: Fixed > Upgrade pre-commit to use latest version of pre-commit and hook > --- > > Key: AIRFLOW-5839 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5839 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5827) Breeze's Build pre-commit should be done as late as possible to optimise experience
[ https://issues.apache.org/jira/browse/AIRFLOW-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5827. --- Fix Version/s: 1.10.7 Resolution: Fixed > Breeze's Build pre-commit should be done as late as possible to optimise > experience > --- > > Key: AIRFLOW-5827 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5827 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5842) Switch to Buster base images
[ https://issues.apache.org/jira/browse/AIRFLOW-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5842: - Assignee: Jarek Potiuk > Switch to Buster base images > > > Key: AIRFLOW-5842 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5842 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5831) Add production image support
[ https://issues.apache.org/jira/browse/AIRFLOW-5831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5831: - Assignee: Jarek Potiuk > Add production image support > > > Key: AIRFLOW-5831 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5831 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > Production image should be build automatically -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5828) Build scripts should be simplified
[ https://issues.apache.org/jira/browse/AIRFLOW-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5828: - Assignee: Jarek Potiuk > Build scripts should be simplified > -- > > Key: AIRFLOW-5828 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5828 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > Builds scripts are unncessarily complex because of hooks/builds were > initially done for Dockerhub only and then adapted to Travis CI. We can > simplify the scripts by moving everything to the scripts/ci to separate bash > functions and make hooks/build use the functions. > Separating everything to functions will make the scripts easy to maintain as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5827) Breeze's Build pre-commit should be done as late as possible to optimise experience
[ https://issues.apache.org/jira/browse/AIRFLOW-5827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5827: - Assignee: Jarek Potiuk > Breeze's Build pre-commit should be done as late as possible to optimise > experience > --- > > Key: AIRFLOW-5827 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5827 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5826) Breeze output should be auto-generated in BREEZE.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5826. --- Fix Version/s: 1.10.7 Resolution: Fixed > Breeze output should be auto-generated in BREEZE.rst > > > Key: AIRFLOW-5826 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5826 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (AIRFLOW-5826) Breeze output should be auto-generated in BREEZE.rst
[ https://issues.apache.org/jira/browse/AIRFLOW-5826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reassigned AIRFLOW-5826: - Assignee: Jarek Potiuk > Breeze output should be auto-generated in BREEZE.rst > > > Key: AIRFLOW-5826 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5826 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5842) Switch to Buster base images
Jarek Potiuk created AIRFLOW-5842: - Summary: Switch to Buster base images Key: AIRFLOW-5842 URL: https://issues.apache.org/jira/browse/AIRFLOW-5842 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 1.10.6, 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5887) User can be removed from CI images
Jarek Potiuk created AIRFLOW-5887: - Summary: User can be removed from CI images Key: AIRFLOW-5887 URL: https://issues.apache.org/jira/browse/AIRFLOW-5887 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 1.10.6, 2.0.0 Reporter: Jarek Potiuk The AIRFLOW_USER is not needed any more in CI images. It will be needed in Prod images but it will be implemented differently there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5888) Use psycopg2-binary package
[ https://issues.apache.org/jira/browse/AIRFLOW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5888. --- Resolution: Fixed > Use psycopg2-binary package > --- > > Key: AIRFLOW-5888 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5888 > Project: Apache Airflow > Issue Type: Improvement > Components: dependencies >Affects Versions: 1.10.4, 1.10.5, 1.10.6 >Reporter: Matthew Bowden >Assignee: Matthew Bowden >Priority: Major > Fix For: 1.10.7 > > > This requirement was pinned as of PR 5036, as related to a breaking change > with respect to the package namespace. This PR modifies which package to > refer to - prior to 2.8 the package was a binary wheel, but with 2.8+ the > appropriate installation package is {{psycopg2-binary}} > h3. Explanation: > [Full explanation > here|http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/] > For most of psycopg2's history, they had distributed pre-compiled binaries > (or "wheels") which would get installed when you did {{pip install psycopg2}} > (rather than compiling the source code upon installation). At some point > psycopg2 found that there were some segfault issues that were appearing in > the pre-compiled versions. Granted, this could be fixed if you installed by > explicitly saying you wanted to compile from source rather than retrieve the > binary version, but the psycopg2 devs felt that this would be better > addressed by publishing a separate package that was the pre-compiled binary, > while keeping the original package as compile-from-source only. The upshot of > this is that everyone who was used to installing {{psycopg2}} and getting the > pre-compiled binaries was now having to compile the package on install unless > they moved over to {{psycopg2-binary}} instead. The packages are effectively > the same, only the pre-compiled versions are usually more optimized, run > quicker on the systems they're build for, and don't require compilation > dependencies to install. So, in accordance with the warning provided by > psycopg2 ({{UserWarning: The psycopg2 wheel package will be renamed from > release 2.8; in order to keep installing from binary please use "pip install > psycopg2-binary" instead.}}), I've swapping this over to psycopg2-binary. > I've also confirmed that {{psycopg2-binary}} is available from {{2.7.4}} > onwards: > $ pip install psycopg2-binary== Collecting psycopg2-binary== ERROR: Could not > find a version that satisfies the requirement psycopg2-binary== (from > versions: 2.7.4, 2.7.5, 2.7.6, 2.7.6.1, 2.7.7, 2.8, 2.8.1, 2.8.2, 2.8.3, > 2.8.4) ERROR: No matching distribution found {color:#d73a49}for{color} > psycopg2-binary== -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5886: -- Parent: AIRFLOW-5606 Issue Type: Sub-task (was: Bug) > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
Jarek Potiuk created AIRFLOW-5886: - Summary: Copying sources in Dockerfile can be selective Key: AIRFLOW-5886 URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.6, 2.0.0 Reporter: Jarek Potiuk This change will further improve time of rebuilds for docker image when your sources change (very useful in case of building kubernetes image). It adds only directories that are needed (it is synchronised with .dockerignore and local mounts) and in the sequence that reflects frequency of changes. Also pip install is not done again after sources change (there is no point) so the build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5830) Get rid of slim image
[ https://issues.apache.org/jira/browse/AIRFLOW-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5830. --- Fix Version/s: 1.10.7 Resolution: Fixed > Get rid of slim image > - > > Key: AIRFLOW-5830 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5830 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > We can get rid of the slim image and use the ci image everywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5878) Use JobID to monitor statuses when running a Dataflow template
[ https://issues.apache.org/jira/browse/AIRFLOW-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5878. --- Fix Version/s: 2.0.0 Resolution: Fixed > Use JobID to monitor statuses when running a Dataflow template > -- > > Key: AIRFLOW-5878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5878 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5878) Use JobID to monitor statuses when running a Dataflow template
[ https://issues.apache.org/jira/browse/AIRFLOW-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reopened AIRFLOW-5878: --- > Use JobID to monitor statuses when running a Dataflow template > -- > > Key: AIRFLOW-5878 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5878 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5854) Docker logs of Python processes are not visible
[ https://issues.apache.org/jira/browse/AIRFLOW-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5854. --- Fix Version/s: 2.0.0 Resolution: Fixed > Docker logs of Python processes are not visible > --- > > Key: AIRFLOW-5854 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5854 > Project: Apache Airflow > Issue Type: New Feature > Components: logging, operators >Affects Versions: 2.0.0 >Reporter: Akshesh Doshi >Assignee: Akshesh Doshi >Priority: Major > Labels: docker, orchestration, swarm > Fix For: 2.0.0 > > > Reason - [https://stackoverflow.com/a/33027467/3061686] > Fix - Add support for `tty` > ([https://docker-py.readthedocs.io/en/stable/api.html#docker.types.ContainerSpec]) > parameter in Docker Swarm operator (and set to True, by default). > > This is also required to fix > https://issues.apache.org/jira/browse/AIRFLOW-5850. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5854) Docker logs of Python processes are not visible
[ https://issues.apache.org/jira/browse/AIRFLOW-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5854: -- Fix Version/s: (was: 2.0.0) 1.10.7 > Docker logs of Python processes are not visible > --- > > Key: AIRFLOW-5854 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5854 > Project: Apache Airflow > Issue Type: New Feature > Components: logging, operators >Affects Versions: 2.0.0 >Reporter: Akshesh Doshi >Assignee: Akshesh Doshi >Priority: Major > Labels: docker, orchestration, swarm > Fix For: 1.10.7 > > > Reason - [https://stackoverflow.com/a/33027467/3061686] > Fix - Add support for `tty` > ([https://docker-py.readthedocs.io/en/stable/api.html#docker.types.ContainerSpec]) > parameter in Docker Swarm operator (and set to True, by default). > > This is also required to fix > https://issues.apache.org/jira/browse/AIRFLOW-5850. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5782) Migrate AWS Lambda to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5782. --- Fix Version/s: 2.0.0 Resolution: Fixed > Migrate AWS Lambda to /providers/aws [AIP-21] > - > > Key: AIRFLOW-5782 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5782 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Cyril Shcherbin >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5786) Migrate AWS SNS to /providers/aws [AIP-21]
[ https://issues.apache.org/jira/browse/AIRFLOW-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5786. --- Fix Version/s: 2.0.0 Resolution: Fixed > Migrate AWS SNS to /providers/aws [AIP-21] > -- > > Key: AIRFLOW-5786 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5786 > Project: Apache Airflow > Issue Type: Improvement > Components: aws >Affects Versions: 2.0.0 >Reporter: Bas Harenslak >Assignee: Rich Dean >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5803) Update S3Hook import paths
[ https://issues.apache.org/jira/browse/AIRFLOW-5803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5803. --- Resolution: Fixed > Update S3Hook import paths > -- > > Key: AIRFLOW-5803 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5803 > Project: Apache Airflow > Issue Type: Improvement > Components: hooks >Affects Versions: 2.0.0 >Reporter: MinJae Kwon >Assignee: MinJae Kwon >Priority: Minor > Fix For: 2.0.0 > > > Should update all S3Hook import paths with new ones. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5846) Implementing Sqlite hook tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5846. --- Fix Version/s: 2.0.0 Resolution: Fixed > Implementing Sqlite hook tests > -- > > Key: AIRFLOW-5846 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5846 > Project: Apache Airflow > Issue Type: Test > Components: tests >Affects Versions: 1.10.6 >Reporter: Sayed Mohammad Hossein Torabi >Assignee: Sayed Mohammad Hossein Torabi >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5855) Broken ref in custom-operator doc
[ https://issues.apache.org/jira/browse/AIRFLOW-5855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5855. --- Fix Version/s: 1.10.7 Resolution: Fixed > Broken ref in custom-operator doc > - > > Key: AIRFLOW-5855 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5855 > Project: Apache Airflow > Issue Type: Bug > Components: documentation >Affects Versions: 1.10.6 >Reporter: Aviem Zur >Assignee: Aviem Zur >Priority: Minor > Fix For: 1.10.7 > > > While running the example in custom-operator doc I encountered a broken > reference which raised an exception. > [https://airflow.readthedocs.io/en/latest/howto/custom-operator.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5829) Get rid of checklicence image
[ https://issues.apache.org/jira/browse/AIRFLOW-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5829. --- Fix Version/s: 1.10.7 Resolution: Fixed > Get rid of checklicence image > - > > Key: AIRFLOW-5829 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5829 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.5, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > We can get rid of checklicence image and run checklicence checks using the > standard CI image -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5829) Get rid of checklicence image
[ https://issues.apache.org/jira/browse/AIRFLOW-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5829: -- Affects Version/s: 1.10.6 > Get rid of checklicence image > - > > Key: AIRFLOW-5829 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5829 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.5, 1.10.6 >Reporter: Jarek Potiuk >Assignee: Jarek Potiuk >Priority: Major > > We can get rid of checklicence image and run checklicence checks using the > standard CI image -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5862) Build fails on new kubernetes approach
Jarek Potiuk created AIRFLOW-5862: - Summary: Build fails on new kubernetes approach Key: AIRFLOW-5862 URL: https://issues.apache.org/jira/browse/AIRFLOW-5862 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5862) Build fails on new kubernetes approach
[ https://issues.apache.org/jira/browse/AIRFLOW-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk closed AIRFLOW-5862. - > Build fails on new kubernetes approach > -- > > Key: AIRFLOW-5862 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5862 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5862) Build fails on new kubernetes approach
[ https://issues.apache.org/jira/browse/AIRFLOW-5862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5862. --- Resolution: Won't Fix Reverted original AIRFLOW-5704 implementation in order to fix the remaining bugs. > Build fails on new kubernetes approach > -- > > Key: AIRFLOW-5862 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5862 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5853) BigQuery - standardize task_ids in example_dag
[ https://issues.apache.org/jira/browse/AIRFLOW-5853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5853. --- Fix Version/s: 2.0.0 Resolution: Fixed > BigQuery - standardize task_ids in example_dag > -- > > Key: AIRFLOW-5853 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5853 > Project: Apache Airflow > Issue Type: Improvement > Components: examples, gcp >Affects Versions: 1.10.6 >Reporter: Tobiasz Kedzierski >Assignee: Tobiasz Kedzierski >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5838) Make all __init__ pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-5838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5838. --- Fix Version/s: 2.0.0 Resolution: Fixed > Make all __init__ pylint compatible > --- > > Key: AIRFLOW-5838 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5838 > Project: Apache Airflow > Issue Type: Improvement > Components: core, tests >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5852) Make tests.www pylint compatible
[ https://issues.apache.org/jira/browse/AIRFLOW-5852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5852. --- Fix Version/s: 2.0.0 Resolution: Fixed > Make tests.www pylint compatible > > > Key: AIRFLOW-5852 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5852 > Project: Apache Airflow > Issue Type: Improvement > Components: tests >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Minor > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5851) Some urlparse and urlquote imports in V1-10 branch are python3-only
[ https://issues.apache.org/jira/browse/AIRFLOW-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5851: -- Fix Version/s: (was: 1.10.7) > Some urlparse and urlquote imports in V1-10 branch are python3-only > --- > > Key: AIRFLOW-5851 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5851 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, core, operators >Affects Versions: 1.10.4, 1.10.5, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > > Those files were affected: > * airflow/contrib/operators/gcp_cloud_build_operator.py > * airflow/models/connection.py > * airflow/contrib/hooks/databricks_hook.py > * airflow/contrib/operators/s3_to_sftp_operator.py > * airflow/contrib/operators/sftp_to_s3_operator.py > * airflow/sensors/s3_key_sensor.py > * airflow/utils/log/gcs_task_handler.py > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5851) Some urlparse and urlquote imports in V1-10 branch are python3-only
[ https://issues.apache.org/jira/browse/AIRFLOW-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reopened AIRFLOW-5851: --- > Some urlparse and urlquote imports in V1-10 branch are python3-only > --- > > Key: AIRFLOW-5851 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5851 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, core, operators >Affects Versions: 1.10.4, 1.10.5, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > Those files were affected: > * airflow/contrib/operators/gcp_cloud_build_operator.py > * airflow/models/connection.py > * airflow/contrib/hooks/databricks_hook.py > * airflow/contrib/operators/s3_to_sftp_operator.py > * airflow/contrib/operators/sftp_to_s3_operator.py > * airflow/sensors/s3_key_sensor.py > * airflow/utils/log/gcs_task_handler.py > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (AIRFLOW-5851) Some urlparse and urlquote imports in V1-10 branch are python3-only
[ https://issues.apache.org/jira/browse/AIRFLOW-5851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk closed AIRFLOW-5851. - Resolution: Won't Fix Closing this one as we had to withdraw from isort for v1-10-test due to circular commits I have done it once but forgot about it. That reminds me one of the highest voted answer to this StackOverflow question: [https://stackoverflow.com/questions/184618/what-is-the-best-comment-in-source-code-you-have-ever-encountered] {code:java} // // Dear maintainer: // // Once you are done trying to 'optimize' this routine, // and have realized what a terrible mistake that was, // please increment the following counter as a warning // to the next guy: // // total_hours_wasted_here = 42 // {code} > Some urlparse and urlquote imports in V1-10 branch are python3-only > --- > > Key: AIRFLOW-5851 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5851 > Project: Apache Airflow > Issue Type: Bug > Components: contrib, core, operators >Affects Versions: 1.10.4, 1.10.5, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > > Those files were affected: > * airflow/contrib/operators/gcp_cloud_build_operator.py > * airflow/models/connection.py > * airflow/contrib/hooks/databricks_hook.py > * airflow/contrib/operators/s3_to_sftp_operator.py > * airflow/contrib/operators/sftp_to_s3_operator.py > * airflow/sensors/s3_key_sensor.py > * airflow/utils/log/gcs_task_handler.py > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AIRFLOW-4496) Airflow `backfill` fails on pickle thread error when --task_regex used
[ https://issues.apache.org/jira/browse/AIRFLOW-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963010#comment-16963010 ] Jarek Potiuk commented on AIRFLOW-4496: --- Hello [~trevorpburke] - do you still have this problem? I believe it has something to do with the operator you are using in the Dag. Can you share the DAG structure you have? > Airflow `backfill` fails on pickle thread error when --task_regex used > -- > > Key: AIRFLOW-4496 > URL: https://issues.apache.org/jira/browse/AIRFLOW-4496 > Project: Apache Airflow > Issue Type: Bug > Components: DAG >Affects Versions: 1.10.2 > Environment: Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-1075-aws x86_64) > Postgres LocalExecutor >Reporter: Trevor Burke >Priority: Major > > Airflow backfill works properly when used without task_regex, but when I > employ that flog I get the following stack trace: > {code:java} > TypeError: can't pickle _thread.RLock objects > {code} > The command I'm using is: > {code:java} > airflow backfill -s 2019-04-15 -e 2019-05-08 -x -t normalize -i > --reset_dagruns > {code} > {code:python} > interval_args = { > 'owner': 'airflow', > 'depends_on_past': True, > 'start_date': datetime(2019, 4, 15), > 'retries': 2, > 'retry_delay': timedelta(minutes=5), > 'on_failure_callback': send_email > } > interval_dag = DAG('dag_id_redacted', >default_args=interval_args, >schedule_interval='*/15 * * * *', >catchup=True, >user_defined_macros=dict(DBT=DBT),) > {code} > The task flow is basically get data from external API, dump to S3, flatten > for database loading, and load to database. The tasks have been performing > perfectly fine and previous backfills have been successful, but task_regex > has been giving me issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker scripts/image for kind kubernetes tests can be optimised
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Parent: AIRFLOW-5606 Issue Type: Sub-task (was: Improvement) > Docker scripts/image for kind kubernetes tests can be optimised > --- > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be optimised: > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5827) Breeze's Build pre-commit should be done as late as possible to optimise experience
Jarek Potiuk created AIRFLOW-5827: - Summary: Breeze's Build pre-commit should be done as late as possible to optimise experience Key: AIRFLOW-5827 URL: https://issues.apache.org/jira/browse/AIRFLOW-5827 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0, 1.10.6 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5829) Get rid of checklicence image
Jarek Potiuk created AIRFLOW-5829: - Summary: Get rid of checklicence image Key: AIRFLOW-5829 URL: https://issues.apache.org/jira/browse/AIRFLOW-5829 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 1.10.5, 2.0.0 Reporter: Jarek Potiuk We can get rid of checklicence image and run checklicence checks using the standard CI image -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5828) Build scripts shoudl be simplified
Jarek Potiuk created AIRFLOW-5828: - Summary: Build scripts shoudl be simplified Key: AIRFLOW-5828 URL: https://issues.apache.org/jira/browse/AIRFLOW-5828 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk Builds scripts are unncessarily complex because of hooks/builds were initially done for Dockerhub only and then adapted to Travis CI. We can simplify the scripts by moving everything to the scripts/ci to separate bash functions and make hooks/build use the functions. Separating everything to functions will make the scripts easy to maintain as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5828) Build scripts should be simplified
[ https://issues.apache.org/jira/browse/AIRFLOW-5828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5828: -- Summary: Build scripts should be simplified (was: Build scripts shoudl be simplified) > Build scripts should be simplified > -- > > Key: AIRFLOW-5828 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5828 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > Builds scripts are unncessarily complex because of hooks/builds were > initially done for Dockerhub only and then adapted to Travis CI. We can > simplify the scripts by moving everything to the scripts/ci to separate bash > functions and make hooks/build use the functions. > Separating everything to functions will make the scripts easy to maintain as > well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5826) Breeze output should be auto-generated in BREEZE.rst
Jarek Potiuk created AIRFLOW-5826: - Summary: Breeze output should be auto-generated in BREEZE.rst Key: AIRFLOW-5826 URL: https://issues.apache.org/jira/browse/AIRFLOW-5826 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0, 1.10.6 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker scripts/image for kind kubernetes tests can be optimised
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Summary: Docker scripts/image for kind kubernetes tests can be optimised (was: Docker image for kind kubernetes tests can be optimised) > Docker scripts/image for kind kubernetes tests can be optimised > --- > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be optimised: > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch > * -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker scripts/image for kind kubernetes tests can be optimised
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Description: The docker CI image for kind tests can be optimised: * Kubernetes Version and all the installation of docker + kubectl + kind can be moved to ci_install* script * Running kubernetes scripts should be possible from within breeze without special environment * --env breeze switch should be removed * ENV variable should disappear * "bare" environment should be replaced by --no-deps switch was: The docker CI image for kind tests can be optimised: * Kubernetes Version and all the installation of docker + kubectl + kind can be moved to ci_install* script * Running kubernetes scripts should be possible from within breeze without special environment * --env breeze switch should be removed * ENV variable should disappear * "bare" environment should be replaced by --no-deps switch * > Docker scripts/image for kind kubernetes tests can be optimised > --- > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be optimised: > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5704) Docker image for kind kubernetes tests can be optimised
[ https://issues.apache.org/jira/browse/AIRFLOW-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5704: -- Description: The docker CI image for kind tests can be optimised: * Kubernetes Version and all the installation of docker + kubectl + kind can be moved to ci_install* script * Running kubernetes scripts should be possible from within breeze without special environment * --env breeze switch should be removed * ENV variable should disappear * "bare" environment should be replaced by --no-deps switch * was:The docker CI image for kind tests can be optimised a bit - Kubernetes Version should be removed and all the installation of docker + kubectl + kind can be moved to ci_install* script > Docker image for kind kubernetes tests can be optimised > --- > > Key: AIRFLOW-5704 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5704 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > The docker CI image for kind tests can be optimised: > > * Kubernetes Version and all the installation of docker + kubectl + kind can > be moved to ci_install* script > * Running kubernetes scripts should be possible from within breeze without > special environment > * --env breeze switch should be removed > * ENV variable should disappear > * "bare" environment should be replaced by --no-deps switch > * -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5831) Add production image support
Jarek Potiuk created AIRFLOW-5831: - Summary: Add production image support Key: AIRFLOW-5831 URL: https://issues.apache.org/jira/browse/AIRFLOW-5831 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0, 1.10.6 Reporter: Jarek Potiuk Production image should be build automatically -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5830) Get rid of slim image
Jarek Potiuk created AIRFLOW-5830: - Summary: Get rid of slim image Key: AIRFLOW-5830 URL: https://issues.apache.org/jira/browse/AIRFLOW-5830 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0, 1.10.6 Reporter: Jarek Potiuk We can get rid of the slim image and use the ci image everywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5917) Group tests for the Webserver command
[ https://issues.apache.org/jira/browse/AIRFLOW-5917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5917. --- Fix Version/s: 2.0.0 Resolution: Fixed > Group tests for the Webserver command > - > > Key: AIRFLOW-5917 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5917 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5898) alembic command crashes due to typing import path collision
[ https://issues.apache.org/jira/browse/AIRFLOW-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5898. --- Fix Version/s: 2.0.0 Resolution: Fixed > alembic command crashes due to typing import path collision > --- > > Key: AIRFLOW-5898 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5898 > Project: Apache Airflow > Issue Type: New Feature > Components: database >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Trivial > Fix For: 2.0.0 > > > Running alemibc within airflow directory results in the following crash: > > {code:java} > File > "/Users/qph/Envs/airflow/lib/python3.7/site-packages/typing_extensions.py", > line 17, in > from typing import Generic, Callable, TypeVar, Tuple > ImportError: cannot import name 'Generic' from 'typing' > {code} > This is due to typing.py within airlfow directory collides with the builtin > typing module when a command runs inside airflow directory. Renaming it to > `custom_typing.py` fixes the issue. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk reopened AIRFLOW-5886: --- > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5886. --- Resolution: Fixed > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5691) Rewrite Dataproc operators to use python library
[ https://issues.apache.org/jira/browse/AIRFLOW-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5691. --- Fix Version/s: 2.0.0 Resolution: Fixed > Rewrite Dataproc operators to use python library > > > Key: AIRFLOW-5691 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5691 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5934) TRAINING ISSUE! Add extra CC: to the emails sent by Airflow
Jarek Potiuk created AIRFLOW-5934: - Summary: TRAINING ISSUE! Add extra CC: to the emails sent by Airflow Key: AIRFLOW-5934 URL: https://issues.apache.org/jira/browse/AIRFLOW-5934 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk {quote}NOTE! THIS IS A TEST issue that is used in Contributor's documentation to show an example issue for the first time contributors. Please do not implement it :). ! {quote} We need to have a way to add extra CC: to the emails sent by Airflow. # You can see the information about how to configure email here email: [https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration] # Here is the class you should modify: [https://github.com/apache/airflow/blob/master/airflow/utils/email.py] # Tests for email class are in: [https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] # Run the tests in Local Virtualenv: [https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst] # Run the tests in Breeze: [https://github.com/apache/airflow/blob/master/BREEZE.rst] # Make a DRAFT PR as described in [https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions] # Run airflow webserver and scheduler and configure it to send emails and connect to a real SMTP server and test if the email is sent with the CC:'d account -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5934) TRAINING ISSUE! Add extra CC: to the emails sent by Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5934: -- Description: {quote}{color:#FF}NOTE! THIS IS A TEST issue that is used in Contributor's documentation to show an example issue for the first time contributors. Please do not implement it :). !{color} {quote} We need to have a way to add extra CC: to the emails sent by Airflow. # You can see the information about how to configure email here email: [https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration] # Here is the class you should modify: [https://github.com/apache/airflow/blob/master/airflow/utils/email.py] # Tests for email class are in: [https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] # Run the tests in Local Virtualenv: [https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst] # Run the tests in Breeze: [https://github.com/apache/airflow/blob/master/BREEZE.rst] # Make a DRAFT PR as described in [https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions] # Run airflow webserver and scheduler and configure it to send emails and connect to a real SMTP server and test if the email is sent with the CC:'d account was: {quote}NOTE! THIS IS A TEST issue that is used in Contributor's documentation to show an example issue for the first time contributors. Please do not implement it :). ! {quote} We need to have a way to add extra CC: to the emails sent by Airflow. # You can see the information about how to configure email here email: [https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration] # Here is the class you should modify: [https://github.com/apache/airflow/blob/master/airflow/utils/email.py] # Tests for email class are in: [https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] # Run the tests in Local Virtualenv: [https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst] # Run the tests in Breeze: [https://github.com/apache/airflow/blob/master/BREEZE.rst] # Make a DRAFT PR as described in [https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions] # Run airflow webserver and scheduler and configure it to send emails and connect to a real SMTP server and test if the email is sent with the CC:'d account > TRAINING ISSUE! Add extra CC: to the emails sent by Airflow > --- > > Key: AIRFLOW-5934 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5934 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > Labels: training-issue- > > {quote}{color:#FF}NOTE! THIS IS A TEST issue that is used in > Contributor's documentation to show an example issue for the first time > contributors. Please do not implement it :). !{color} > {quote} > We need to have a way to add extra CC: to the emails sent by Airflow. > # You can see the information about how to configure email here email: > [https://airflow.readthedocs.io/en/latest/concepts.html#email-configuration] > # Here is the class you should modify: > [https://github.com/apache/airflow/blob/master/airflow/utils/email.py] > # Tests for email class are in: > [https://github.com/apache/airflow/blob/master/tests/utils/test_email.py] > # Run the tests in Local Virtualenv: > [https://github.com/apache/airflow/blob/master/LOCAL_VIRTUALENV.rst] > # Run the tests in Breeze: > [https://github.com/apache/airflow/blob/master/BREEZE.rst] > # Make a DRAFT PR as described in > [https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contributions] > # Run airflow webserver and scheduler and configure it to send emails and > connect to a real SMTP server and test if the email is sent with the CC:'d > account -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5768) Google Cloud SQL - Don't store ephemeral connection object to database
[ https://issues.apache.org/jira/browse/AIRFLOW-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5768. --- Fix Version/s: 2.0.0 Resolution: Fixed > Google Cloud SQL - Don't store ephemeral connection object to database > -- > > Key: AIRFLOW-5768 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5768 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp >Affects Versions: 1.10.5 >Reporter: Daniel Standish >Assignee: Daniel Standish >Priority: Major > Fix For: 2.0.0 > > > GCP cloud sql operator creates dynamically an ephemeral Connection object. > It persists to metastore during execution and deletes afterward. > This behavior has negative impact on our ability to refactor creds > management. > By not persisting to database, we can also remove some complexity re ensuring > connection is deleted in event of failure, and the tests that go along with > that. > This change requires that we add optional param `connection` to both > MySqlHook and PostgresHook. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5923) Use absolute paths in GCP system tests
[ https://issues.apache.org/jira/browse/AIRFLOW-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5923. --- Fix Version/s: 2.0.0 Resolution: Fixed > Use absolute paths in GCP system tests > -- > > Key: AIRFLOW-5923 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5923 > Project: Apache Airflow > Issue Type: Improvement > Components: gcp, tests >Affects Versions: 1.10.7 >Reporter: Tomasz Urbaszek >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5940) Remove redundant Code in models.dagrun
[ https://issues.apache.org/jira/browse/AIRFLOW-5940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5940. --- Fix Version/s: 2.0.0 Resolution: Resolved > Remove redundant Code in models.dagrun > -- > > Key: AIRFLOW-5940 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5940 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0, 1.10.6 >Reporter: Kaxil Naik >Assignee: Kaxil Naik >Priority: Trivial > Fix For: 2.0.0 > > > The below code in models.dagrun contains redundant code > {code:python} > is_task_in_dag = task is not None > should_restore_task = is_task_in_dag and ti.state == State.REMOVED > {code} > It can be simplified by: > {code:python} > should_restore_task = (task is not None) and ti.state == State.REMOVED > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5936) Allow explicit get_pty in SSHOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-5936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5936. --- Fix Version/s: 2.0.0 Resolution: Fixed > Allow explicit get_pty in SSHOperator > - > > Key: AIRFLOW-5936 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5936 > Project: Apache Airflow > Issue Type: Improvement > Components: operators >Affects Versions: 1.10.6 >Reporter: Zikun Zhu >Priority: Major > Fix For: 2.0.0 > > > Currently when execution_timeout is reached for an SSHOperator task, the ssh > connection will be closed but the remote process continues to run. In many > scenarios, users might want the process to be killed upon task timeout. > Giving users an explicit get_pty option achieves this goal. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6007) Doc check checks provider/ instead of providers/ packages
Jarek Potiuk created AIRFLOW-6007: - Summary: Doc check checks provider/ instead of providers/ packages Key: AIRFLOW-6007 URL: https://issues.apache.org/jira/browse/AIRFLOW-6007 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6003) Cyclic imports detected by pylint randomly
Jarek Potiuk created AIRFLOW-6003: - Summary: Cyclic imports detected by pylint randomly Key: AIRFLOW-6003 URL: https://issues.apache.org/jira/browse/AIRFLOW-6003 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk There are cyclic imports detected seemingly randomly by pylint checks when some of the PRs are run in CI: An example of such pylint problem: [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status] * Module airflow.utils.log.json_formatter 1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_generator) (cyclic-import) 1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow -> airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_launcher) (cyclic-import) 1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator) (cyclic-import) The problem is that airflow's __init__ contains a few convenience imports (AirflowException, Executors etc.) but it also imports a number of packages (for example kubernetes_executor) that in turn import the airflow package objects - for example airflow.Executor. This leads to cyclic imports if you import first the executors before airflow. Similar problem happens with executor.__init__.py containing class "Executors" imported by all executors but at the same time some of the executors (for example KubernetesExecutor) import the very same Executor class. This might happen in pylint checks in pre-commit because they split a number of files they process between the multiple threads you have at your machine and sometimes it might happen that the files are imported in different order. As a solution, the executors "list" should be moved to a separate module and a number of classes (such as AirflowException) should be moved out from __init__ package and imported from where they belong (so for example AirflowException should be universally imported from airflow.exceptions). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
[ https://issues.apache.org/jira/browse/AIRFLOW-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6004: -- Description: See the description in https://issues.apache.org/jira/browse/AIRFLOW-6003 (was: See the description in ) > Untangle "Executors" class from potentially cyclic import > - > > Key: AIRFLOW-6004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6004 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > See the description in https://issues.apache.org/jira/browse/AIRFLOW-6003 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
Jarek Potiuk created AIRFLOW-6004: - Summary: Untangle "Executors" class from potentially cyclic import Key: AIRFLOW-6004 URL: https://issues.apache.org/jira/browse/AIRFLOW-6004 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6004) Untangle "Executors" class from potentially cyclic import
[ https://issues.apache.org/jira/browse/AIRFLOW-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6004: -- Description: See the description in > Untangle "Executors" class from potentially cyclic import > - > > Key: AIRFLOW-6004 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6004 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > See the description in -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6006) Move DAG out of airflow package imports
Jarek Potiuk created AIRFLOW-6006: - Summary: Move DAG out of airflow package imports Key: AIRFLOW-6006 URL: https://issues.apache.org/jira/browse/AIRFLOW-6006 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6005) Remove all imports of AirflowException directly from airflow package
Jarek Potiuk created AIRFLOW-6005: - Summary: Remove all imports of AirflowException directly from airflow package Key: AIRFLOW-6005 URL: https://issues.apache.org/jira/browse/AIRFLOW-6005 Project: Apache Airflow Issue Type: Sub-task Components: ci Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-3489) PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly
[ https://issues.apache.org/jira/browse/AIRFLOW-3489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-3489. --- Fix Version/s: 2.0.0 Resolution: Fixed > PostgresToGoogleCloudStorageOperator doesn't handle PostgreSQL json properly > - > > Key: AIRFLOW-3489 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3489 > Project: Apache Airflow > Issue Type: Bug > Components: gcp >Reporter: Duan Shiqiang >Priority: Major > Labels: bigquery > Fix For: 2.0.0 > > > PostgresToGoogleCloudStorageOperator saves json data (postgres json or jsonb) > as native python types (i.e. dictionary) to gcs new line separated json data. > But it generates bigquery schema for that field as data type string which > won't work if user want to import the gcs data into bigquery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-6003) Cyclic imports detected by pylint randomly
[ https://issues.apache.org/jira/browse/AIRFLOW-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-6003: -- Description: There are cyclic imports detected seemingly randomly by pylint checks when some of the PRs are run in CI: An example of such pylint problem: [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status] {code:java} * Module airflow.utils.log.json_formatter airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_generator) (cyclic-import) airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow -> airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_launcher) (cyclic-import) airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator) (cyclic-import) {code} The problem is that airflow's __init__ contains a few convenience imports (AirflowException, Executors etc.) but it also imports a number of packages (for example kubernetes_executor) that in turn import the airflow package objects - for example airflow.Executor. This leads to cyclic imports if you import first the executors before airflow. Similar problem happens with executor.__init__.py containing class "Executors" imported by all executors but at the same time some of the executors (for example KubernetesExecutor) import the very same Executor class. This might happen in pylint checks in pre-commit because they split a number of files they process between the multiple threads you have at your machine and sometimes it might happen that the files are imported in different order. As a solution, the executors "list" should be moved to a separate module and a number of classes (such as AirflowException) should be moved out from __init__ package and imported from where they belong (so for example AirflowException should be universally imported from airflow.exceptions). was: There are cyclic imports detected seemingly randomly by pylint checks when some of the PRs are run in CI: An example of such pylint problem: [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status] * Module airflow.utils.log.json_formatter 1492airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_generator) (cyclic-import) 1493airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow -> airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.pod_launcher) (cyclic-import) 1494airflow/utils/log/json_formatter.py:1:0: R0401: Cyclic import (airflow.executors -> airflow.executors.kubernetes_executor -> airflow.kubernetes.worker_configuration -> airflow.kubernetes.pod_generator) (cyclic-import) The problem is that airflow's __init__ contains a few convenience imports (AirflowException, Executors etc.) but it also imports a number of packages (for example kubernetes_executor) that in turn import the airflow package objects - for example airflow.Executor. This leads to cyclic imports if you import first the executors before airflow. Similar problem happens with executor.__init__.py containing class "Executors" imported by all executors but at the same time some of the executors (for example KubernetesExecutor) import the very same Executor class. This might happen in pylint checks in pre-commit because they split a number of files they process between the multiple threads you have at your machine and sometimes it might happen that the files are imported in different order. As a solution, the executors "list" should be moved to a separate module and a number of classes (such as AirflowException) should be moved out from __init__ package and imported from where they belong (so for example AirflowException should be universally imported from airflow.exceptions). > Cyclic imports detected by pylint randomly > -- > > Key: AIRFLOW-6003 > URL: https://issues.apache.org/jira/browse/AIRFLOW-6003 > Project: Apache Airflow > Issue Type: Bug > Components: ci >Affects Versions: 2.0.0 >Reporter: Jarek Potiuk >Priority: Major > > There are cyclic imports detected seemingly randomly by pylint checks when > some of the PRs are run in CI: > An example of such pylint problem: > [https://travis-ci.org/apache/airflow/jobs/612096100?utm_medium=notification_source=github_status] > > > {code:java} > * Module airflow.utils.log.json_formatter >
[jira] [Created] (AIRFLOW-6009) Only kubernetes tests need travis_wait
Jarek Potiuk created AIRFLOW-6009: - Summary: Only kubernetes tests need travis_wait Key: AIRFLOW-6009 URL: https://issues.apache.org/jira/browse/AIRFLOW-6009 Project: Apache Airflow Issue Type: Bug Components: ci Affects Versions: 1.10.6, 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6010) The "from airflow import settings" is used far too often
Jarek Potiuk created AIRFLOW-6010: - Summary: The "from airflow import settings" is used far too often Key: AIRFLOW-6010 URL: https://issues.apache.org/jira/browse/AIRFLOW-6010 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk In many places "from airflow import settings" is used only to get the latest set value of one or few settings or when single method from settings is going to be called. Settings loads a lot of dependencies on its own and having it imported at the top of the file often causes many transitive/implicit dependencies which might cause circular imports. Therefore in many places the settings should be imported as local imports. This causes lasy-loading of settings in case of tests but also it makes sure that even if the values in settings are overridden by some other initialisation, we will always use latest version of settings. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5824) Add a new hook and operator for AWS DataSync
[ https://issues.apache.org/jira/browse/AIRFLOW-5824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5824. --- Fix Version/s: 2.0.0 Resolution: Fixed > Add a new hook and operator for AWS DataSync > > > Key: AIRFLOW-5824 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5824 > Project: Apache Airflow > Issue Type: New Feature > Components: aws >Affects Versions: 1.10.5 >Reporter: Bjorn Olsen >Assignee: Bjorn Olsen >Priority: Minor > Fix For: 2.0.0 > > > I'm working on some code to add a hook and operator for AWS DataSync task > executions. > Just creating a Jira ticket for visibility -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5894) Group tests for the Dags command
[ https://issues.apache.org/jira/browse/AIRFLOW-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5894. --- Fix Version/s: 2.0.0 Resolution: Fixed > Group tests for the Dags command > > > Key: AIRFLOW-5894 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5894 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5901) Group tests for the tasks and variables command
[ https://issues.apache.org/jira/browse/AIRFLOW-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5901. --- Fix Version/s: 2.0.0 Resolution: Fixed > Group tests for the tasks and variables command > --- > > Key: AIRFLOW-5901 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5901 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5887) User can be removed from CI images
[ https://issues.apache.org/jira/browse/AIRFLOW-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5887. --- Fix Version/s: 1.10.7 Resolution: Fixed > User can be removed from CI images > -- > > Key: AIRFLOW-5887 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5887 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > The AIRFLOW_USER is not needed any more in CI images. It will be needed in > Prod images but it will be implemented differently there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5886: -- Fix Version/s: (was: 2.0.0) 1.10.7 > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5886) Copying sources in Dockerfile can be selective
[ https://issues.apache.org/jira/browse/AIRFLOW-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5886. --- Fix Version/s: 2.0.0 Resolution: Fixed > Copying sources in Dockerfile can be selective > -- > > Key: AIRFLOW-5886 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5886 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 2.0.0 > > > This change will further improve time of rebuilds for docker image when your > sources change (very useful in case of building kubernetes image). It adds > only directories that are needed (it is synchronised with .dockerignore and > local mounts) and in the sequence that reflects frequency of changes. Also > pip install is not done again after sources change (there is no point) so the > build is much faster when only sources or test file change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5897) Allow setting -1 as pool slots value in webserver
[ https://issues.apache.org/jira/browse/AIRFLOW-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5897. --- Fix Version/s: 1.10.7 Resolution: Fixed > Allow setting -1 as pool slots value in webserver > - > > Key: AIRFLOW-5897 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5897 > Project: Apache Airflow > Issue Type: Improvement > Components: webserver >Affects Versions: 2.0.0 >Reporter: Kevin Yang >Assignee: Kevin Yang >Priority: Major > Fix For: 1.10.7 > > > Given https://github.com/apache/airflow/pull/6520, we should also allow -1 as > pool slots value in the webserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5876) Fetch all Dataflow jobs during searching
[ https://issues.apache.org/jira/browse/AIRFLOW-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5876. --- Fix Version/s: 2.0.0 Resolution: Fixed > Fetch all Dataflow jobs during searching > > > Key: AIRFLOW-5876 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5876 > Project: Apache Airflow > Issue Type: Sub-task > Components: gcp >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5882) Running dependency should not be ignorable
[ https://issues.apache.org/jira/browse/AIRFLOW-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5882. --- Fix Version/s: 2.0.0 Resolution: Fixed > Running dependency should not be ignorable > -- > > Key: AIRFLOW-5882 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5882 > Project: Apache Airflow > Issue Type: Improvement > Components: worker >Affects Versions: 1.10.6 >Reporter: Alex Guziel >Assignee: Alex Guziel >Priority: Major > Fix For: 2.0.0 > > > This ti_state dep should not be ignoreable if the task is in the running > state as it would allow double-run in the "ideal" situation and in the > current situation it causes both tasks to die. Due to celery visibility > timeout, the task will always kill itself eventually -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5896) Move email stuff from tests/core.py
[ https://issues.apache.org/jira/browse/AIRFLOW-5896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5896. --- Fix Version/s: 2.0.0 Resolution: Fixed > Move email stuff from tests/core.py > --- > > Key: AIRFLOW-5896 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5896 > Project: Apache Airflow > Issue Type: Bug > Components: core >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5900) heartbeat_callback for local_task_job can be more efficient
[ https://issues.apache.org/jira/browse/AIRFLOW-5900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5900. --- Fix Version/s: 2.0.0 Resolution: Fixed > heartbeat_callback for local_task_job can be more efficient > --- > > Key: AIRFLOW-5900 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5900 > Project: Apache Airflow > Issue Type: Improvement > Components: core >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Trivial > Fix For: 2.0.0 > > > local_task_job's heartbeat_callback function is issuing unnecessary system > calls when they are not needed. We can move those calls into the if block to > make it more efficient. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5892) BashOperator does not create temporary shell script
[ https://issues.apache.org/jira/browse/AIRFLOW-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5892. --- Fix Version/s: 2.0.0 Resolution: Fixed > BashOperator does not create temporary shell script > --- > > Key: AIRFLOW-5892 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5892 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > > Bash Operator needlessly creates temporary file to run it. It can all be done > in python without creating file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (AIRFLOW-5892) BashOperator does not create temporary shell script
[ https://issues.apache.org/jira/browse/AIRFLOW-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk updated AIRFLOW-5892: -- Description: Bash Operator needlessly creates temporary file to run it. It can all be done in python without creating file. > BashOperator does not create temporary shell script > --- > > Key: AIRFLOW-5892 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5892 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > > Bash Operator needlessly creates temporary file to run it. It can all be done > in python without creating file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5885) List of tests should not be generated during Docker build
[ https://issues.apache.org/jira/browse/AIRFLOW-5885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5885. --- Fix Version/s: 1.10.7 Resolution: Fixed > List of tests should not be generated during Docker build > - > > Key: AIRFLOW-5885 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5885 > Project: Apache Airflow > Issue Type: Sub-task > Components: ci >Affects Versions: 2.0.0, 1.10.6 >Reporter: Jarek Potiuk >Priority: Major > Fix For: 1.10.7 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5899) dmypy still reports error
[ https://issues.apache.org/jira/browse/AIRFLOW-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5899. --- Fix Version/s: 2.0.0 Resolution: Fixed > dmypy still reports error > - > > Key: AIRFLOW-5899 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5899 > Project: Apache Airflow > Issue Type: Improvement > Components: ci >Affects Versions: 2.0.0 >Reporter: QP Hou >Assignee: QP Hou >Priority: Trivial > Fix For: 2.0.0 > > > dmypy is still reporting error, would be nice to get those fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5066) Usage of FieldRef in Kubernetes env execution
[ https://issues.apache.org/jira/browse/AIRFLOW-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5066. --- Fix Version/s: 1.10.7 Resolution: Fixed > Usage of FieldRef in Kubernetes env execution > - > > Key: AIRFLOW-5066 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5066 > Project: Apache Airflow > Issue Type: Bug > Components: executors >Affects Versions: 1.10.4, 2.0.0, 1.10.5 >Reporter: Julien Surloppe >Priority: Minor > Fix For: 1.10.7 > > > In Kubernetes, dynamic pod metadata can be set as environment variables and > then used as substitution in other environment variables. > Currently in Airflow, the pod request creation set those variables after the > ones that can use substitutions so them doen't do it. > Example usage is: > * define a variable SPARK_LOCAL_IP using the pod metadata status.podIP > * using this variable in Spark options as > -Dspark.driver.host=$(SPARK_LOCAL_IP) > I will do a PR to fix this behavior. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5873) KubernetesPodOperator fixes and test
[ https://issues.apache.org/jira/browse/AIRFLOW-5873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5873. --- Resolution: Fixed > KubernetesPodOperator fixes and test > > > Key: AIRFLOW-5873 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5873 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.10.6 >Reporter: David de L. >Assignee: David de L. >Priority: Major > Fix For: 2.0.0, 1.10.7 > > > * `KubernetesPodOperator` kwarg `resources` is erroneously passed to > `base_operator`, instead should only go to `PodGenerator`. The two have > different syntax. (both on `master` and `v1-10-test` branches) > * `resources` passed to PodGenerator [should be > `k8s.V1ResourceRequirements`](https://github.com/kubernetes-client/python/blob/3739db034d34b035bc8141740e9073c84ab192c0/kubernetes/client/models/v1_container.py#L45), > which is now handled in `KubernetesPodOperator` > * `kubernetes/pod.py`: `Resources` does not have `__slots__` so accepts > arbitrary values in `setattr` (present on `v1-10-test` branch > [https://github.com/apache/airflow/blame/v1-10-test/airflow/contrib/operators/kubernetes_pod_operator.py#L166]) > * `v1-10-test` is behind `master` with KubernetesPodOperator fixes and > refactors > ** e.g. move kubernetes folder one level up from `/contrib` > [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L21] > ** fix `xcom_push` to `do_xcom_push` > [https://github.com/apache/airflow/blame/4dd24a2c595d4042ffe745aed947eaaea6abb652/airflow/contrib/operators/kubernetes_pod_operator.py#L90] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-5919) Group tests for the Users/Roles/permission commands
[ https://issues.apache.org/jira/browse/AIRFLOW-5919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-5919. --- Fix Version/s: 2.0.0 Resolution: Fixed > Group tests for the Users/Roles/permission commands > --- > > Key: AIRFLOW-5919 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5919 > Project: Apache Airflow > Issue Type: Sub-task > Components: cli >Affects Versions: 1.10.6 >Reporter: Kamil Bregula >Priority: Major > Fix For: 2.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (AIRFLOW-3333) New features enable transferring of files or data from GCS to a SFTP remote path and SFTP to GCS path.
[ https://issues.apache.org/jira/browse/AIRFLOW-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Potiuk resolved AIRFLOW-. --- Fix Version/s: 2.0.0 Resolution: Fixed > New features enable transferring of files or data from GCS to a SFTP remote > path and SFTP to GCS path. > --- > > Key: AIRFLOW- > URL: https://issues.apache.org/jira/browse/AIRFLOW- > Project: Apache Airflow > Issue Type: New Feature > Components: contrib, gcp >Reporter: Pulin Pathneja >Priority: Major > Fix For: 2.0.0 > > > New features enable transferring of files or data from GCS(Google Cloud > Storage) to a SFTP remote path and SFTP to GCS(Google Cloud Storage) path. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5960) Change import paths in "databricks" module
Jarek Potiuk created AIRFLOW-5960: - Summary: Change import paths in "databricks" module Key: AIRFLOW-5960 URL: https://issues.apache.org/jira/browse/AIRFLOW-5960 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5956) Change import paths in "apache/spark" modules
Jarek Potiuk created AIRFLOW-5956: - Summary: Change import paths in "apache/spark" modules Key: AIRFLOW-5956 URL: https://issues.apache.org/jira/browse/AIRFLOW-5956 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5958) Change import paths in "mysql" modules
Jarek Potiuk created AIRFLOW-5958: - Summary: Change import paths in "mysql" modules Key: AIRFLOW-5958 URL: https://issues.apache.org/jira/browse/AIRFLOW-5958 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5959) Change import paths in "jira" modules
Jarek Potiuk created AIRFLOW-5959: - Summary: Change import paths in "jira" modules Key: AIRFLOW-5959 URL: https://issues.apache.org/jira/browse/AIRFLOW-5959 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5955) Change import paths in "apache/pinot" modules
Jarek Potiuk created AIRFLOW-5955: - Summary: Change import paths in "apache/pinot" modules Key: AIRFLOW-5955 URL: https://issues.apache.org/jira/browse/AIRFLOW-5955 Project: Apache Airflow Issue Type: Sub-task Components: core Affects Versions: 2.0.0 Reporter: Jarek Potiuk -- This message was sent by Atlassian Jira (v8.3.4#803005)