[jira] [Created] (AIRFLOW-3106) Validate connection when saving connection
Bas Harenslak created AIRFLOW-3106: -- Summary: Validate connection when saving connection Key: AIRFLOW-3106 URL: https://issues.apache.org/jira/browse/AIRFLOW-3106 Project: Apache Airflow Issue Type: Improvement Reporter: Bas Harenslak I've encountered failures in DAG runs at various occasions due to invalid connection credentials, or a domain was unreachable from the Airflow instance. It'd be nice to validate a connection when saving it, to directly know if a given connection can be made or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3237) Refactor example DAGs
Bas Harenslak created AIRFLOW-3237: -- Summary: Refactor example DAGs Key: AIRFLOW-3237 URL: https://issues.apache.org/jira/browse/AIRFLOW-3237 Project: Apache Airflow Issue Type: Improvement Reporter: Bas Harenslak I think it's important to have the example DAGs reflect good programming standards and how to write clean and concise Airflow code. Therefore, refactor the example DAGs, which mostly involves: * Replace set_upstream() and set_downstream() by the bitshift operators, since they exist since Airflow 1.8 and everybody uses those now. * Optimise imports (grouping by stdlib/3rd party/own and alphabetic ordering). * Place function calls with arguments on single line if line is short enough, or put each function argument on a new line, but don't mix. * Passing list of tasks when setting dependencies instead of making multiple set_up/down-stream calls. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3237) Refactor example DAGs
[ https://issues.apache.org/jira/browse/AIRFLOW-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-3237: --- Description: I think it's important to have the example DAGs reflect good programming standards and how to write clean and concise Airflow code. Therefore, refactor the example DAGs, which mostly involves: * Replace set_upstream() and set_downstream() by the bitshift operators, since they exist since Airflow 1.8 and everybody uses those now. * Optimise imports (grouping by stdlib/3rd party/own and alphabetic ordering). * Place function calls with arguments on single line if line is short enough, or put each function argument on a new line, but don't mix. * Passing list of tasks when setting dependencies instead of making multiple set_up/down-stream calls. * Switch upstream dependencies into downstream dependencies since these are more logical to read IMO. was: I think it's important to have the example DAGs reflect good programming standards and how to write clean and concise Airflow code. Therefore, refactor the example DAGs, which mostly involves: * Replace set_upstream() and set_downstream() by the bitshift operators, since they exist since Airflow 1.8 and everybody uses those now. * Optimise imports (grouping by stdlib/3rd party/own and alphabetic ordering). * Place function calls with arguments on single line if line is short enough, or put each function argument on a new line, but don't mix. * Passing list of tasks when setting dependencies instead of making multiple set_up/down-stream calls. > Refactor example DAGs > - > > Key: AIRFLOW-3237 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3237 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Priority: Major > > I think it's important to have the example DAGs reflect good programming > standards and how to write clean and concise Airflow code. Therefore, > refactor the example DAGs, which mostly involves: > * Replace set_upstream() and set_downstream() by the bitshift operators, > since they exist since Airflow 1.8 and everybody uses those now. > * Optimise imports (grouping by stdlib/3rd party/own and alphabetic ordering). > * Place function calls with arguments on single line if line is short enough, > or put each function argument on a new line, but don't mix. > * Passing list of tasks when setting dependencies instead of making multiple > set_up/down-stream calls. > * Switch upstream dependencies into downstream dependencies since these are > more logical to read IMO. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3375) Support returning multiple tasks with BranchPythonOperator
Bas Harenslak created AIRFLOW-3375: -- Summary: Support returning multiple tasks with BranchPythonOperator Key: AIRFLOW-3375 URL: https://issues.apache.org/jira/browse/AIRFLOW-3375 Project: Apache Airflow Issue Type: Improvement Reporter: Bas Harenslak I hit a case where I'm using the BranchPythonOperator and want to branch to multiple tasks, so I added support to returning a list of task ids. Both a single task id (string type) and list of task ids are supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-3375) Support returning multiple tasks with BranchPythonOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-3375: --- Description: I hit a case where I'm using the BranchPythonOperator and want to branch to multiple tasks, so I added support to returning a list of task ids. Both a single task id (string type) and list of task ids are supported. PR: https://github.com/apache/incubator-airflow/pull/4215 was: I hit a case where I'm using the BranchPythonOperator and want to branch to multiple tasks, so I added support to returning a list of task ids. Both a single task id (string type) and list of task ids are supported. > Support returning multiple tasks with BranchPythonOperator > -- > > Key: AIRFLOW-3375 > URL: https://issues.apache.org/jira/browse/AIRFLOW-3375 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Bas Harenslak >Priority: Major > > I hit a case where I'm using the BranchPythonOperator and want to branch to > multiple tasks, so I added support to returning a list of task ids. > Both a single task id (string type) and list of task ids are supported. > PR: https://github.com/apache/incubator-airflow/pull/4215 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-3556) Add a "cross join" function for setting dependencies between two lists of tasks
Bas Harenslak created AIRFLOW-3556:
--
Summary: Add a "cross join" function for setting dependencies
between two lists of tasks
Key: AIRFLOW-3556
URL: https://issues.apache.org/jira/browse/AIRFLOW-3556
Project: Apache Airflow
Issue Type: New Feature
Reporter: Bas Harenslak
Similar to airflow.utils.helpers.chain(), it would be useful to have a helper
function that sets downstream dependencies in a cross join fashion between two
lists of tasks.
For example:
{code}
cross_downstream(from_tasks=[t1, t2, t3], to_tasks=[t4, t5, t6])
Sets dependencies:
t1 --> t4
\ /
t2 -X> t5
/ \
t3 --> t6
Equivalent to:
t1.set_downstream(t4)
t1.set_downstream(t5)
t1.set_downstream(t6)
t2.set_downstream(t4)
t2.set_downstream(t5)
t2.set_downstream(t6)
t3.set_downstream(t4)
t3.set_downstream(t5)
t3.set_downstream(t6){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (AIRFLOW-3557) Various typos
Bas Harenslak created AIRFLOW-3557: -- Summary: Various typos Key: AIRFLOW-3557 URL: https://issues.apache.org/jira/browse/AIRFLOW-3557 Project: Apache Airflow Issue Type: Bug Reporter: Bas Harenslak Fix various typos, checked with [misspell|https://github.com/client9/misspell]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2439) Official Docker image
Bas Harenslak created AIRFLOW-2439: -- Summary: Official Docker image Key: AIRFLOW-2439 URL: https://issues.apache.org/jira/browse/AIRFLOW-2439 Project: Apache Airflow Issue Type: New Feature Reporter: Bas Harenslak Assignee: Bas Harenslak I think we need an official Airflow Docker image, especially with the creation of the [KubernetesExecutor|https://issues.apache.org/jira/browse/AIRFLOW-1314]. There's lots of 3rd party Airflow Docker images, but now that we have example Kubernetes resources which require an image to be specified, it would benefit to have an official Docker image (instead of first having to build your own). There's already a [Dockerfile|https://github.com/apache/incubator-airflow/blob/master/scripts/ci/kubernetes/docker/Dockerfile], we could integrate a docker push to Docker Hub in the Travis CI. To request an official Apache Airflow Docker Hub account, we need to file an official repository PR at the docker-library repo. I would like to set up and file the request, and there might be some back-and-forth discussion before the proposal is accepted. [More info|https://docs.docker.com/docker-hub/official_repos/#how-do-i-create-a-new-official-repository] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2439) Official Docker image
[ https://issues.apache.org/jira/browse/AIRFLOW-2439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2439: --- Description: I think we need an official Airflow Docker image, especially with the creation of the KubernetesExecutor. There's lots of 3rd party Airflow Docker images, but now that we have example Kubernetes resources which require an image to be specified, it would benefit to have an official Docker image (instead of first having to build your own). To request an official Apache Airflow Docker Hub account, we need to file an official repository PR at the docker-library repo. I would like to set up and file the request, and there might be some back-and-forth discussion before the proposal is accepted. [More info|https://docs.docker.com/docker-hub/official_repos/#how-do-i-create-a-new-official-repository] I like the setup Flink has done by creating a separate docker-flink account in which the docker-library repos are forked, and a repo with scripts e.g. for generating the docker library manifest: [https://github.com/docker-flink]. was: I think we need an official Airflow Docker image, especially with the creation of the [KubernetesExecutor|https://issues.apache.org/jira/browse/AIRFLOW-1314]. There's lots of 3rd party Airflow Docker images, but now that we have example Kubernetes resources which require an image to be specified, it would benefit to have an official Docker image (instead of first having to build your own). There's already a [Dockerfile|https://github.com/apache/incubator-airflow/blob/master/scripts/ci/kubernetes/docker/Dockerfile], we could integrate a docker push to Docker Hub in the Travis CI. To request an official Apache Airflow Docker Hub account, we need to file an official repository PR at the docker-library repo. I would like to set up and file the request, and there might be some back-and-forth discussion before the proposal is accepted. [More info|https://docs.docker.com/docker-hub/official_repos/#how-do-i-create-a-new-official-repository] > Official Docker image > - > > Key: AIRFLOW-2439 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2439 > Project: Apache Airflow > Issue Type: New Feature >Reporter: Bas Harenslak >Assignee: Bas Harenslak >Priority: Major > > I think we need an official Airflow Docker image, especially with the > creation of the KubernetesExecutor. There's lots of 3rd party Airflow Docker > images, but now that we have example Kubernetes resources which require an > image to be specified, it would benefit to have an official Docker image > (instead of first having to build your own). > To request an official Apache Airflow Docker Hub account, we need to file an > official repository PR at the docker-library repo. I would like to set up and > file the request, and there might be some back-and-forth discussion before > the proposal is accepted. [More > info|https://docs.docker.com/docker-hub/official_repos/#how-do-i-create-a-new-official-repository] > I like the setup Flink has done by creating a separate docker-flink account > in which the docker-library repos are forked, and a repo with scripts e.g. > for generating the docker library manifest: [https://github.com/docker-flink]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2475) Add a "go to triggered DAG" button when using TriggerDagRunOperator
Bas Harenslak created AIRFLOW-2475: -- Summary: Add a "go to triggered DAG" button when using TriggerDagRunOperator Key: AIRFLOW-2475 URL: https://issues.apache.org/jira/browse/AIRFLOW-2475 Project: Apache Airflow Issue Type: Improvement Components: ui Reporter: Bas Harenslak Assignee: Bas Harenslak Attachments: image-2018-05-16-22-37-15-891.png It would be useful to have a button for going to the triggered DAG view when using the TriggerDagRunOperator, instead of reading code to know which DAG was triggered. I've implemented this and the modal now looks like this when opening a TriggerDagRunOperator. PR coming. !image-2018-05-16-22-37-15-891.png|width=345,height=253! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (AIRFLOW-2505) Add state removed to the legend
Bas Harenslak created AIRFLOW-2505: -- Summary: Add state removed to the legend Key: AIRFLOW-2505 URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 Project: Apache Airflow Issue Type: Improvement Components: ui Reporter: Bas Harenslak Attachments: image-2018-05-22-08-54-48-137.png, image-2018-05-22-08-55-52-245.png There's currently no legend item for removed tasks. I think we should add it. !image-2018-05-22-08-55-52-245.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: (was: image-2018-05-22-08-55-52-245.png) > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-08-57-05-525.png, > image-2018-05-22-08-57-35-362.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-08-57-35-362.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: image-2018-05-22-08-57-05-525.png > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-08-55-52-245.png, > image-2018-05-22-08-57-05-525.png, image-2018-05-22-08-57-35-362.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-08-55-52-245.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: image-2018-05-22-08-57-35-362.png > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-08-55-52-245.png, > image-2018-05-22-08-57-05-525.png, image-2018-05-22-08-57-35-362.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-08-55-52-245.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: (was: image-2018-05-22-08-54-48-137.png) > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-08-57-05-525.png, > image-2018-05-22-08-57-35-362.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-08-57-35-362.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Description: There's currently no legend item for removed tasks. I think we should add it. !image-2018-05-22-08-57-35-362.png! was: There's currently no legend item for removed tasks. I think we should add it. !image-2018-05-22-08-55-52-245.png! > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-08-55-52-245.png, > image-2018-05-22-08-57-05-525.png, image-2018-05-22-08-57-35-362.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-08-57-35-362.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: (was: image-2018-05-22-08-57-05-525.png) > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-08-57-35-362.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-08-57-35-362.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: image-2018-05-22-09-09-04-754.png Description: There's currently no legend item for removed tasks. I think we should add it. !image-2018-05-22-09-09-04-754.png! was: There's currently no legend item for removed tasks. I think we should add it. !image-2018-05-22-08-57-35-362.png! > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-09-09-04-754.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-09-09-04-754.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AIRFLOW-2505) Add state removed to the legend
[ https://issues.apache.org/jira/browse/AIRFLOW-2505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bas Harenslak updated AIRFLOW-2505: --- Attachment: (was: image-2018-05-22-08-57-35-362.png) > Add state removed to the legend > --- > > Key: AIRFLOW-2505 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2505 > Project: Apache Airflow > Issue Type: Improvement > Components: ui >Reporter: Bas Harenslak >Priority: Minor > Attachments: image-2018-05-22-09-09-04-754.png > > > There's currently no legend item for removed tasks. I think we should add it. > !image-2018-05-22-09-09-04-754.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
