[jira] [Created] (AIRFLOW-6652) Set is_delete_operator_pod=True by default in KubernetesPodOperator()
Mathew Wicks created AIRFLOW-6652: - Summary: Set is_delete_operator_pod=True by default in KubernetesPodOperator() Key: AIRFLOW-6652 URL: https://issues.apache.org/jira/browse/AIRFLOW-6652 Project: Apache Airflow Issue Type: Improvement Components: operators Affects Versions: 1.10.7 Reporter: Mathew Wicks As we are not far from a major release (Airflow 2.0) could we consider changing the default behaviour of the *is_delete_operator_pod* argument in *KubernetesPodOperator*() to *True, r*ather than False? For most usecases, it is not desirable that the pod continues running independent of the airflow task, and having an unsafe default is bad practice. Thoughts? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-6827) PROPOSAL: Implement a run_pool concept for DagRuns
Mathew Wicks created AIRFLOW-6827: - Summary: PROPOSAL: Implement a run_pool concept for DagRuns Key: AIRFLOW-6827 URL: https://issues.apache.org/jira/browse/AIRFLOW-6827 Project: Apache Airflow Issue Type: New Feature Components: DagRun, scheduler Affects Versions: 1.10.9 Reporter: Mathew Wicks Currently we have a concept of resource pools for tasks, where there can be at most N instances of a task running at once. However, no such concept exists for DagRuns. For example, you might have a requirement that two DAGs (with different dag_id) cannot run at the same time, so you might define a run_pool for these DAGs with size 1. Meaning only 1 DagRun may be in the RUNNING state at a time. I am interested in hearing everyone's thoughts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (AIRFLOW-5033) Cannot install apache-airflow[kerberos] on Python 3
Mathew Wicks created AIRFLOW-5033: - Summary: Cannot install apache-airflow[kerberos] on Python 3 Key: AIRFLOW-5033 URL: https://issues.apache.org/jira/browse/AIRFLOW-5033 Project: Apache Airflow Issue Type: Bug Components: build Affects Versions: 1.10.3 Reporter: Mathew Wicks Currently you cannot install apache-airflow[kerberos] with Python 3. *Steps to reproduce: (For Ubuntu)* # Ensure you have the required build dependencies, _*sudo apt install libsasl2-dev libkrb5-dev*_ # Create a new conda environment. _*conda create -n airflow_36 python=3.6*_ # Activate the environment: _*conda activate airflow_36*_ # Attempt to install _*pip install apache-airflow[kerberos]*_ # **fails** The error is caused by trying to install python-krbV as a depednancy of snakebite[kerberos], as python-krbV does not support Python 3. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-5033) Cannot install apache-airflow[kerberos] on Python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892213#comment-16892213 ] Mathew Wicks commented on AIRFLOW-5033: --- [~jackjack10] Yes, with _*pip install apache-airflow[kerberos]==1.10.4rc3*_ the same error occurs. > Cannot install apache-airflow[kerberos] on Python 3 > --- > > Key: AIRFLOW-5033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5033 > Project: Apache Airflow > Issue Type: Bug > Components: build >Affects Versions: 1.10.3 >Reporter: Mathew Wicks >Priority: Major > > Currently you cannot install apache-airflow[kerberos] with Python 3. > *Steps to reproduce: (For Ubuntu)* > # Ensure you have the required build dependencies, _*sudo apt install > libsasl2-dev libkrb5-dev*_ > # Create a new conda environment. _*conda create -n airflow_36 python=3.6*_ > # Activate the environment: _*conda activate airflow_36*_ > # Attempt to install _*pip install apache-airflow[kerberos]*_ > # **fails** > The error is caused by trying to install python-krbV as a depednancy of > snakebite[kerberos], as python-krbV does not support Python 3. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (AIRFLOW-5033) Cannot install apache-airflow[kerberos] on Python 3
[ https://issues.apache.org/jira/browse/AIRFLOW-5033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892452#comment-16892452 ] Mathew Wicks commented on AIRFLOW-5033: --- [~jackjack10], the issue doesn't happen in CI because we manually exclude it: [https://github.com/apache/airflow/blob/1.10.3/setup.py#L269] > Cannot install apache-airflow[kerberos] on Python 3 > --- > > Key: AIRFLOW-5033 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5033 > Project: Apache Airflow > Issue Type: Bug > Components: build >Affects Versions: 1.10.3 >Reporter: Mathew Wicks >Priority: Major > > Currently you cannot install apache-airflow[kerberos] with Python 3. > *Steps to reproduce: (For Ubuntu)* > # Ensure you have the required build dependencies, _*sudo apt install > libsasl2-dev libkrb5-dev*_ > # Create a new conda environment. _*conda create -n airflow_36 python=3.6*_ > # Activate the environment: _*conda activate airflow_36*_ > # Attempt to install _*pip install apache-airflow[kerberos]*_ > # **fails** > The error is caused by trying to install python-krbV as a depednancy of > snakebite[kerberos], as python-krbV does not support Python 3. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (AIRFLOW-5261) RBAC can't use LDAP -- 'python-ldap package is missing from setup.py
Mathew Wicks created AIRFLOW-5261: - Summary: RBAC can't use LDAP -- 'python-ldap package is missing from setup.py Key: AIRFLOW-5261 URL: https://issues.apache.org/jira/browse/AIRFLOW-5261 Project: Apache Airflow Issue Type: Bug Components: webserver Affects Versions: 1.10.4 Reporter: Mathew Wicks Currently with airflow 1.10.4, if you setup a clean python 3 environment, and install airflow with: {code:python} pip install airflow[ldap] {code} You will be missing a required package for LDAP to work with the new RBAC, giving: {code:java} Traceback (most recent call last): File "PYTHON_HOME/lib/python3.6/site-packages/flask_appbuilder/security/manager.py", line 725, in auth_user_ldap ldap.set_option(ldap.OPT_X_TLS_CACERTFILE, self.auth_ldap_tls_cacertfile) ModuleNotFoundError: No module named 'ldap' {code} This is because the 'ldap' tag in setup.py doesn't include the 'python-ldap' package, which is needed for 'import ldap' to succeed. The change would need to be made here: [https://github.com/apache/airflow/blob/1.10.4/setup.py#L229] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (AIRFLOW-5261) RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py
[ https://issues.apache.org/jira/browse/AIRFLOW-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mathew Wicks updated AIRFLOW-5261: -- Summary: RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py (was: RBAC can't use LDAP -- 'python-ldap package is missing from setup.py) > RBAC can't use LDAP -- 'python-ldap' package is missing from setup.py > - > > Key: AIRFLOW-5261 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5261 > Project: Apache Airflow > Issue Type: Bug > Components: webserver >Affects Versions: 1.10.4 >Reporter: Mathew Wicks >Priority: Major > > Currently with airflow 1.10.4, if you setup a clean python 3 environment, and > install airflow with: > {code:python} > pip install airflow[ldap] > {code} > You will be missing a required package for LDAP to work with the new RBAC, > giving: > {code:java} > Traceback (most recent call last): > File > "PYTHON_HOME/lib/python3.6/site-packages/flask_appbuilder/security/manager.py", > line 725, in auth_user_ldap > ldap.set_option(ldap.OPT_X_TLS_CACERTFILE, self.auth_ldap_tls_cacertfile) > ModuleNotFoundError: No module named 'ldap' > {code} > This is because the 'ldap' tag in setup.py doesn't include the 'python-ldap' > package, which is needed for 'import ldap' to succeed. > The change would need to be made here: > [https://github.com/apache/airflow/blob/1.10.4/setup.py#L229] -- This message was sent by Atlassian Jira (v8.3.2#803003)