[jira] [Resolved] (AIRFLOW-1736) Add HotelQuickly to Who Uses Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jetsada Machom resolved AIRFLOW-1736. - Resolution: Done > Add HotelQuickly to Who Uses Airflow > > > Key: AIRFLOW-1736 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1736 > Project: Apache Airflow > Issue Type: Task >Reporter: Jetsada Machom >Assignee: Jetsada Machom >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (AIRFLOW-1736) Add HotelQuickly to Who Uses Airflow
[ https://issues.apache.org/jira/browse/AIRFLOW-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-1736 started by Jetsada Machom. --- > Add HotelQuickly to Who Uses Airflow > > > Key: AIRFLOW-1736 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1736 > Project: Apache Airflow > Issue Type: Task >Reporter: Jetsada Machom >Assignee: Jetsada Machom >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (AIRFLOW-1310) Kubernetes execute operator
[ https://issues.apache.org/jira/browse/AIRFLOW-1310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Ishikawa closed AIRFLOW-1310. Resolution: Duplicate > Kubernetes execute operator > --- > > Key: AIRFLOW-1310 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1310 > Project: Apache Airflow > Issue Type: New Feature > Components: operators >Reporter: Yu Ishikawa >Assignee: Dennis Docter > > We should support operators for Kubernetes to execute containers remotely. > - Create a Kubernetes cluster > - Execute PODs > - https://github.com/kubernetes-incubator/client-python -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1736) Add HotelQuickly to Who Uses Airflow
Jetsada Machom created AIRFLOW-1736: --- Summary: Add HotelQuickly to Who Uses Airflow Key: AIRFLOW-1736 URL: https://issues.apache.org/jira/browse/AIRFLOW-1736 Project: Apache Airflow Issue Type: Task Reporter: Jetsada Machom Assignee: Jetsada Machom Priority: Trivial -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1735) Log files do not show up for unscheduled dags
[ https://issues.apache.org/jira/browse/AIRFLOW-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1735: - Description: I created a new dag with a schedule of {{@once}} and left it unscheduled. I ran different tasks a few times from the UI but the logs never showed up. Instead I'd see messages like the following: {code} *** Log file isn't local. *** Fetching here: http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21 *** Failed to fetch log file from worker. {code} There is a log file for this task, but it seems to be at a slightly different path on the machine: {code} (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00 -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00 {code} And indeed, when I change the URL to WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request&dag_id=DAG_ID&execution_date=2017-06-21T00:00:00, I see the actual log: {code} *** Reading local log. [2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from ~/analytics_kochava_import_dag.py [2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava app IDs ... [2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0 {code} I've added a screen shot of it below. *FIX:* I have noticed in www/view.py the issue is not have the time component 00:00:00 and we can get that quite easily if it doesnt have it. You can see in this PR I made, but basically if we set {code} dttm = dateutil.parser.parse(request.args.get('execution_date')) execution_date = dttm.isoformat() {code} This will fix it. was: I created a new dag with a schedule of {{@once}} and left it unscheduled. I ran different tasks a few times from the UI but the logs never showed up. Instead I'd see messages like the following: {code} *** Log file isn't local. *** Fetching here: http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21 *** Failed to fetch log file from worker. {code} There is a log file for this task, but it seems to be at a slightly different path on the machine: {code} (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00 -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00 {code} And indeed, when I change the URL to WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request&dag_id=DAG_ID&execution_date=2017-06-21T00:00:00, I see the actual log: {code} *** Reading local log. [2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from ~/analytics_kochava_import_dag.py [2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava app IDs ... [2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0 {code} I've added a screen shot of it below. *FIX:* I have noticed in www/view.py the issue is not have the time component 00:00:00 and we can get that quite easily if it doesnt have it. You can see in this PR I made. > Log files do not show up for unscheduled dags > - > > Key: AIRFLOW-1735 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1735 > Project: Apache Airflow > Issue Type: Bug >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: www > Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png > > > I created a new dag with a schedule of {{@once}} and left it unscheduled. I > ran different tasks a few times from the UI but the logs never showed up. > Instead I'd see messages like the following: > {code} > *** Log file isn't local. > *** Fetching here: > http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21 > *** Failed to fetch log file from worker. > {code} > There is a log file for this task, but it seems to be at a slightly different > path on the machine: > {code} > (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth > ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00 > -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 > ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00 > {code} > And indeed, when I change the URL to > WEB_SERVER/admin/airflow/log?task_id=TASK_ID_
[jira] [Created] (AIRFLOW-1735) Log files do not show up for unscheduled dags
Ace Haidrey created AIRFLOW-1735: Summary: Log files do not show up for unscheduled dags Key: AIRFLOW-1735 URL: https://issues.apache.org/jira/browse/AIRFLOW-1735 Project: Apache Airflow Issue Type: Bug Reporter: Ace Haidrey Assignee: Ace Haidrey Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png I created a new dag with a schedule of {{@once}} and left it unscheduled. I ran different tasks a few times from the UI but the logs never showed up. Instead I'd see messages like the following: {code} *** Log file isn't local. *** Fetching here: http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21 *** Failed to fetch log file from worker. {code} There is a log file for this task, but it seems to be at a slightly different path on the machine: {code} (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00 -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00 {code} And indeed, when I change the URL to WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request&dag_id=DAG_ID&execution_date=2017-06-21T00:00:00, I see the actual log: {code} *** Reading local log. [2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from ~/analytics_kochava_import_dag.py [2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava app IDs ... [2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0 {code} I've added a screen shot of it below. *FIX:* I have noticed in www/view.py the issue is not have the time component 00:00:00 and we can get that quite easily if it doesnt have it. You can see in this PR I made. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
[ https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1734: - Description: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes assign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 Here is my PR with change by change explanation: https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131 was: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes assign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Sqoop Operator contains logic errors & needs options to pass more sqoop > options > --- > > Key: AIRFLOW-1734 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > > After taking a look at the Sqoop Operator I have found a number of errors > with it. I will describe them in depth on the PR I will make, but it includes > assign wrong parameters, not hiding the users raw password in the logs > (though a method to do that had been created in the hook, wrong if > conditions, and the inability to add other sqoop options that would make > sense to have the option to pass those. > I have made some remarks in past commits about it too in the case I'm > misinterpreting but regardless there are definitely errors. > https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Here is my PR with change by change explanation: > https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
[ https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ace Haidrey updated AIRFLOW-1734: - Description: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes assign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 was: After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes passign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 > Sqoop Operator contains logic errors & needs options to pass more sqoop > options > --- > > Key: AIRFLOW-1734 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 > Project: Apache Airflow > Issue Type: Bug > Components: contrib >Reporter: Ace Haidrey >Assignee: Ace Haidrey > Labels: patch > > After taking a look at the Sqoop Operator I have found a number of errors > with it. I will describe them in depth on the PR I will make, but it includes > assign wrong parameters, not hiding the users raw password in the logs > (though a method to do that had been created in the hook, wrong if > conditions, and the inability to add other sqoop options that would make > sense to have the option to pass those. > I have made some remarks in past commits about it too in the case I'm > misinterpreting but regardless there are definitely errors. > https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options
Ace Haidrey created AIRFLOW-1734: Summary: Sqoop Operator contains logic errors & needs options to pass more sqoop options Key: AIRFLOW-1734 URL: https://issues.apache.org/jira/browse/AIRFLOW-1734 Project: Apache Airflow Issue Type: Bug Components: contrib Reporter: Ace Haidrey Assignee: Ace Haidrey After taking a look at the Sqoop Operator I have found a number of errors with it. I will describe them in depth on the PR I will make, but it includes passign wrong parameters, not hiding the users raw password in the logs (though a method to do that had been created in the hook, wrong if conditions, and the inability to add other sqoop options that would make sense to have the option to pass those. I have made some remarks in past commits about it too in the case I'm misinterpreting but regardless there are definitely errors. https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1733) BashOperator readline can hang
Dan Davydov created AIRFLOW-1733: Summary: BashOperator readline can hang Key: AIRFLOW-1733 URL: https://issues.apache.org/jira/browse/AIRFLOW-1733 Project: Apache Airflow Issue Type: Bug Reporter: Dan Davydov If the child bash process dies for some reason, the BashOperator will hang since we use a blocking readline call to get output from the child. Instead we should read asynchronously or poll the child bash process like described in the solutions here: https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1732) Improve Dataflow Hook Logging
Trevor Edwards created AIRFLOW-1732: --- Summary: Improve Dataflow Hook Logging Key: AIRFLOW-1732 URL: https://issues.apache.org/jira/browse/AIRFLOW-1732 Project: Apache Airflow Issue Type: Bug Reporter: Trevor Edwards Assignee: Trevor Edwards Logging output for Dataflow hook could be a bit more useful. Namely: # Log the command that is used for opening a dataflow subprocess # If the dataflow subprocess experiences an error, log that error at warning (instead of debug) Currently, errors are extremely opaque, only showing the exit code. The command used is unknown and the error is logged, but it is at the debug level which makes it difficult to find. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts
[ https://issues.apache.org/jira/browse/AIRFLOW-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209921#comment-16209921 ] ASF subversion and git services commented on AIRFLOW-1698: -- Commit 00dd06eb0a387f85c1f16ce3aa17fc36473990bc in incubator-airflow's branch refs/heads/v1-9-test from [~maxime.beauche...@apache.org] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=00dd06e ] [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd In the very early days, the Airflow scheduler needed to be restarted every so often to take new DAG_FOLDERS mutations into account properly. This is no longer required. Closes #2677 from mistercrunch/scheduler_runs (cherry picked from commit b464d23a6d9799b75ae53b3f858e779a48115fed) Signed-off-by: Bolke de Bruin > Remove confusing SCHEDULER_RUNS env var from systemd scripts > > > Key: AIRFLOW-1698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1698 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.9.0 > > > In the very early days, the Airflow scheduler needed to be restarted > every so often to take new DAG_FOLDERS mutations into account properly. > This is no longer needed, so I'm removing artifacts in the > systemd scripts. > I'm unclear whether daemon scripts belong here. I can see how it's > useful to have this somewhere, > but it sits on the edge of what should be in the repo and can get > out-of-sync. > As a result it ends up creating more confusion than help. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts
[ https://issues.apache.org/jira/browse/AIRFLOW-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-1698. - Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2677 [https://github.com/apache/incubator-airflow/pull/2677] > Remove confusing SCHEDULER_RUNS env var from systemd scripts > > > Key: AIRFLOW-1698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1698 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.9.0 > > > In the very early days, the Airflow scheduler needed to be restarted > every so often to take new DAG_FOLDERS mutations into account properly. > This is no longer needed, so I'm removing artifacts in the > systemd scripts. > I'm unclear whether daemon scripts belong here. I can see how it's > useful to have this somewhere, > but it sits on the edge of what should be in the repo and can get > out-of-sync. > As a result it ends up creating more confusion than help. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts
[ https://issues.apache.org/jira/browse/AIRFLOW-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209920#comment-16209920 ] ASF subversion and git services commented on AIRFLOW-1698: -- Commit b464d23a6d9799b75ae53b3f858e779a48115fed in incubator-airflow's branch refs/heads/master from [~maxime.beauche...@apache.org] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b464d23 ] [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd In the very early days, the Airflow scheduler needed to be restarted every so often to take new DAG_FOLDERS mutations into account properly. This is no longer required. Closes #2677 from mistercrunch/scheduler_runs > Remove confusing SCHEDULER_RUNS env var from systemd scripts > > > Key: AIRFLOW-1698 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1698 > Project: Apache Airflow > Issue Type: Bug >Reporter: Maxime Beauchemin > Fix For: 1.9.0 > > > In the very early days, the Airflow scheduler needed to be restarted > every so often to take new DAG_FOLDERS mutations into account properly. > This is no longer needed, so I'm removing artifacts in the > systemd scripts. > I'm unclear whether daemon scripts belong here. I can see how it's > useful to have this somewhere, > but it sits on the edge of what should be in the repo and can get > out-of-sync. > As a result it ends up creating more confusion than help. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd
Repository: incubator-airflow Updated Branches: refs/heads/master 6110139a8 -> b464d23a6 [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd In the very early days, the Airflow scheduler needed to be restarted every so often to take new DAG_FOLDERS mutations into account properly. This is no longer required. Closes #2677 from mistercrunch/scheduler_runs Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b464d23a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b464d23a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b464d23a Branch: refs/heads/master Commit: b464d23a6d9799b75ae53b3f858e779a48115fed Parents: 6110139 Author: Maxime Beauchemin Authored: Wed Oct 18 21:55:54 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:55:57 2017 +0200 -- scripts/systemd/airflow | 4 +--- scripts/systemd/airflow-scheduler.service | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b464d23a/scripts/systemd/airflow -- diff --git a/scripts/systemd/airflow b/scripts/systemd/airflow index 5317548..bb2156b 100644 --- a/scripts/systemd/airflow +++ b/scripts/systemd/airflow @@ -16,6 +16,4 @@ # # AIRFLOW_CONFIG= # AIRFLOW_HOME= -# -# required setting, 0 sets it to unlimited. Scheduler will get restart after every X runs -SCHEDULER_RUNS=5 \ No newline at end of file + http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b464d23a/scripts/systemd/airflow-scheduler.service -- diff --git a/scripts/systemd/airflow-scheduler.service b/scripts/systemd/airflow-scheduler.service index 95229d6..29daa39 100644 --- a/scripts/systemd/airflow-scheduler.service +++ b/scripts/systemd/airflow-scheduler.service @@ -21,9 +21,9 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS} +ExecStart=/bin/airflow scheduler Restart=always RestartSec=5s [Install] -WantedBy=multi-user.target \ No newline at end of file +WantedBy=multi-user.target
incubator-airflow git commit: [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd
Repository: incubator-airflow Updated Branches: refs/heads/v1-9-test c6e5ae7c5 -> 00dd06eb0 [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd In the very early days, the Airflow scheduler needed to be restarted every so often to take new DAG_FOLDERS mutations into account properly. This is no longer required. Closes #2677 from mistercrunch/scheduler_runs (cherry picked from commit b464d23a6d9799b75ae53b3f858e779a48115fed) Signed-off-by: Bolke de Bruin Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/00dd06eb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/00dd06eb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/00dd06eb Branch: refs/heads/v1-9-test Commit: 00dd06eb0a387f85c1f16ce3aa17fc36473990bc Parents: c6e5ae7 Author: Maxime Beauchemin Authored: Wed Oct 18 21:55:54 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:57:31 2017 +0200 -- scripts/systemd/airflow | 4 +--- scripts/systemd/airflow-scheduler.service | 4 ++-- 2 files changed, 3 insertions(+), 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/00dd06eb/scripts/systemd/airflow -- diff --git a/scripts/systemd/airflow b/scripts/systemd/airflow index 5317548..bb2156b 100644 --- a/scripts/systemd/airflow +++ b/scripts/systemd/airflow @@ -16,6 +16,4 @@ # # AIRFLOW_CONFIG= # AIRFLOW_HOME= -# -# required setting, 0 sets it to unlimited. Scheduler will get restart after every X runs -SCHEDULER_RUNS=5 \ No newline at end of file + http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/00dd06eb/scripts/systemd/airflow-scheduler.service -- diff --git a/scripts/systemd/airflow-scheduler.service b/scripts/systemd/airflow-scheduler.service index 95229d6..29daa39 100644 --- a/scripts/systemd/airflow-scheduler.service +++ b/scripts/systemd/airflow-scheduler.service @@ -21,9 +21,9 @@ EnvironmentFile=/etc/sysconfig/airflow User=airflow Group=airflow Type=simple -ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS} +ExecStart=/bin/airflow scheduler Restart=always RestartSec=5s [Install] -WantedBy=multi-user.target \ No newline at end of file +WantedBy=multi-user.target
[jira] [Resolved] (AIRFLOW-1694) Hive Hooks: Python 3 does not have an `itertools.izip` function
[ https://issues.apache.org/jira/browse/AIRFLOW-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-1694. - Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2674 [https://github.com/apache/incubator-airflow/pull/2674] > Hive Hooks: Python 3 does not have an `itertools.izip` function > --- > > Key: AIRFLOW-1694 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1694 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Reporter: Yati > Fix For: 1.9.0 > > > In `hooks/hive_hooks.py`, we have a use of `itertools.izip()`, which would > not work in Python 3, but the builtin `zip()` returns an iterator. > `six.moves` can be used from either Python 2/3 to import such objects that > have been moved around. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1694] Stop using itertools.izip
Repository: incubator-airflow Updated Branches: refs/heads/master 2da485681 -> 6110139a8 [AIRFLOW-1694] Stop using itertools.izip Itertools.zip does not exist in Python 3. Closes #2674 from yati-sagade/fix-py3-zip Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6110139a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6110139a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6110139a Branch: refs/heads/master Commit: 6110139a8fd7069c87e3c99b3976865576180f67 Parents: 2da4856 Author: Yati Sagade Authored: Wed Oct 18 21:53:33 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:53:36 2017 +0200 -- airflow/hooks/hive_hooks.py | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6110139a/airflow/hooks/hive_hooks.py -- diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py index 7c73491..a8f9c8f 100644 --- a/airflow/hooks/hive_hooks.py +++ b/airflow/hooks/hive_hooks.py @@ -14,7 +14,7 @@ # from __future__ import print_function -from builtins import zip +from six.moves import zip from past.builtins import basestring import unicodecsv as csv @@ -144,11 +144,9 @@ class HiveCliHook(BaseHook): if not d: return [] return as_flattened_list( -itertools.izip( -["-hiveconf"] * len(d), -["{}={}".format(k, v) for k, v in d.items()] -) -) +zip(["-hiveconf"] * len(d), +["{}={}".format(k, v) for k, v in d.items()]) +) def run_cli(self, hql, schema=None, verbose=True, hive_conf=None): """
incubator-airflow git commit: [AIRFLOW-1694] Stop using itertools.izip
Repository: incubator-airflow Updated Branches: refs/heads/v1-9-test 31805e836 -> c6e5ae7c5 [AIRFLOW-1694] Stop using itertools.izip Itertools.zip does not exist in Python 3. Closes #2674 from yati-sagade/fix-py3-zip (cherry picked from commit 6110139a8fd7069c87e3c99b3976865576180f67) Signed-off-by: Bolke de Bruin Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c6e5ae7c Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c6e5ae7c Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c6e5ae7c Branch: refs/heads/v1-9-test Commit: c6e5ae7c57224ceb52cbf8d15b3362afd3616053 Parents: 31805e8 Author: Yati Sagade Authored: Wed Oct 18 21:53:33 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:54:24 2017 +0200 -- airflow/hooks/hive_hooks.py | 10 -- 1 file changed, 4 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c6e5ae7c/airflow/hooks/hive_hooks.py -- diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py index 7c73491..a8f9c8f 100644 --- a/airflow/hooks/hive_hooks.py +++ b/airflow/hooks/hive_hooks.py @@ -14,7 +14,7 @@ # from __future__ import print_function -from builtins import zip +from six.moves import zip from past.builtins import basestring import unicodecsv as csv @@ -144,11 +144,9 @@ class HiveCliHook(BaseHook): if not d: return [] return as_flattened_list( -itertools.izip( -["-hiveconf"] * len(d), -["{}={}".format(k, v) for k, v in d.items()] -) -) +zip(["-hiveconf"] * len(d), +["{}={}".format(k, v) for k, v in d.items()]) +) def run_cli(self, hql, schema=None, verbose=True, hive_conf=None): """
[jira] [Commented] (AIRFLOW-1694) Hive Hooks: Python 3 does not have an `itertools.izip` function
[ https://issues.apache.org/jira/browse/AIRFLOW-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209915#comment-16209915 ] ASF subversion and git services commented on AIRFLOW-1694: -- Commit 6110139a8fd7069c87e3c99b3976865576180f67 in incubator-airflow's branch refs/heads/master from Yati Sagade [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6110139 ] [AIRFLOW-1694] Stop using itertools.izip Itertools.zip does not exist in Python 3. Closes #2674 from yati-sagade/fix-py3-zip > Hive Hooks: Python 3 does not have an `itertools.izip` function > --- > > Key: AIRFLOW-1694 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1694 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Reporter: Yati > Fix For: 1.9.0 > > > In `hooks/hive_hooks.py`, we have a use of `itertools.izip()`, which would > not work in Python 3, but the builtin `zip()` returns an iterator. > `six.moves` can be used from either Python 2/3 to import such objects that > have been moved around. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1694) Hive Hooks: Python 3 does not have an `itertools.izip` function
[ https://issues.apache.org/jira/browse/AIRFLOW-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209916#comment-16209916 ] ASF subversion and git services commented on AIRFLOW-1694: -- Commit c6e5ae7c57224ceb52cbf8d15b3362afd3616053 in incubator-airflow's branch refs/heads/v1-9-test from Yati Sagade [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=c6e5ae7 ] [AIRFLOW-1694] Stop using itertools.izip Itertools.zip does not exist in Python 3. Closes #2674 from yati-sagade/fix-py3-zip (cherry picked from commit 6110139a8fd7069c87e3c99b3976865576180f67) Signed-off-by: Bolke de Bruin > Hive Hooks: Python 3 does not have an `itertools.izip` function > --- > > Key: AIRFLOW-1694 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1694 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Reporter: Yati > Fix For: 1.9.0 > > > In `hooks/hive_hooks.py`, we have a use of `itertools.izip()`, which would > not work in Python 3, but the builtin `zip()` returns an iterator. > `six.moves` can be used from either Python 2/3 to import such objects that > have been moved around. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1692) Master cannot be checked out on windows
[ https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209907#comment-16209907 ] ASF subversion and git services commented on AIRFLOW-1692: -- Commit 31805e836a04700b5881705f58b33f32f2c8742a in incubator-airflow's branch refs/heads/v1-9-test from root [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=31805e8 ] [AIRFLOW-1692] Change test_views filename to support Windows Closes #2673 from NielsZeilemaker/AIRFLOW-1692 (cherry picked from commit 2da48568116f7382258e2678e20ee0f9573d653c) Signed-off-by: Bolke de Bruin > Master cannot be checked out on windows > --- > > Key: AIRFLOW-1692 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1692 > Project: Apache Airflow > Issue Type: Bug >Reporter: Niels Zeilemaker >Assignee: Niels Zeilemaker > Fix For: 1.9.0 > > > This file: > /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log > Cannot be created on windows due to the colons in the directory name -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1692) Master cannot be checked out on windows
[ https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bolke de Bruin resolved AIRFLOW-1692. - Resolution: Fixed Fix Version/s: 1.9.0 Issue resolved by pull request #2673 [https://github.com/apache/incubator-airflow/pull/2673] > Master cannot be checked out on windows > --- > > Key: AIRFLOW-1692 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1692 > Project: Apache Airflow > Issue Type: Bug >Reporter: Niels Zeilemaker >Assignee: Niels Zeilemaker > Fix For: 1.9.0 > > > This file: > /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log > Cannot be created on windows due to the colons in the directory name -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1692) Master cannot be checked out on windows
[ https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209906#comment-16209906 ] ASF subversion and git services commented on AIRFLOW-1692: -- Commit 31805e836a04700b5881705f58b33f32f2c8742a in incubator-airflow's branch refs/heads/v1-9-test from root [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=31805e8 ] [AIRFLOW-1692] Change test_views filename to support Windows Closes #2673 from NielsZeilemaker/AIRFLOW-1692 (cherry picked from commit 2da48568116f7382258e2678e20ee0f9573d653c) Signed-off-by: Bolke de Bruin > Master cannot be checked out on windows > --- > > Key: AIRFLOW-1692 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1692 > Project: Apache Airflow > Issue Type: Bug >Reporter: Niels Zeilemaker >Assignee: Niels Zeilemaker > Fix For: 1.9.0 > > > This file: > /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log > Cannot be created on windows due to the colons in the directory name -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1692) Master cannot be checked out on windows
[ https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209905#comment-16209905 ] ASF subversion and git services commented on AIRFLOW-1692: -- Commit 2da48568116f7382258e2678e20ee0f9573d653c in incubator-airflow's branch refs/heads/master from root [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2da4856 ] [AIRFLOW-1692] Change test_views filename to support Windows Closes #2673 from NielsZeilemaker/AIRFLOW-1692 > Master cannot be checked out on windows > --- > > Key: AIRFLOW-1692 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1692 > Project: Apache Airflow > Issue Type: Bug >Reporter: Niels Zeilemaker >Assignee: Niels Zeilemaker > > This file: > /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log > Cannot be created on windows due to the colons in the directory name -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1692] Change test_views filename to support Windows
Repository: incubator-airflow Updated Branches: refs/heads/v1-9-test fcc3c2f6e -> 31805e836 [AIRFLOW-1692] Change test_views filename to support Windows Closes #2673 from NielsZeilemaker/AIRFLOW-1692 (cherry picked from commit 2da48568116f7382258e2678e20ee0f9573d653c) Signed-off-by: Bolke de Bruin Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/31805e83 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/31805e83 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/31805e83 Branch: refs/heads/v1-9-test Commit: 31805e836a04700b5881705f58b33f32f2c8742a Parents: fcc3c2f Author: root Authored: Wed Oct 18 21:49:58 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:50:54 2017 +0200 -- .../task_for_testing_log_view/2017-09-01T00.00.00/1.log| 1 + .../task_for_testing_log_view/2017-09-01T00:00:00/1.log| 1 - tests/www/test_views.py| 2 ++ 3 files changed, 3 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/31805e83/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log -- diff --git a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log new file mode 100644 index 000..bc10ef7 --- /dev/null +++ b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log @@ -0,0 +1 @@ +Log for testing. http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/31805e83/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log -- diff --git a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log deleted file mode 100644 index bc10ef7..000 --- a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log +++ /dev/null @@ -1 +0,0 @@ -Log for testing. http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/31805e83/tests/www/test_views.py -- diff --git a/tests/www/test_views.py b/tests/www/test_views.py index 553e7cc..4931487 100644 --- a/tests/www/test_views.py +++ b/tests/www/test_views.py @@ -328,6 +328,8 @@ class TestLogView(unittest.TestCase): current_dir = os.path.dirname(os.path.abspath(__file__)) logging_config['handlers']['file.task']['base_log_folder'] = os.path.normpath( os.path.join(current_dir, 'test_logs')) +logging_config['handlers']['file.task']['filename_template'] = \ +'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts | replace(":", ".") }}/{{ try_number }}.log' # Write the custom logging configuration to a file self.settings_folder = tempfile.mkdtemp()
incubator-airflow git commit: [AIRFLOW-1692] Change test_views filename to support Windows
Repository: incubator-airflow Updated Branches: refs/heads/master 97a2393f6 -> 2da485681 [AIRFLOW-1692] Change test_views filename to support Windows Closes #2673 from NielsZeilemaker/AIRFLOW-1692 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2da48568 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2da48568 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2da48568 Branch: refs/heads/master Commit: 2da48568116f7382258e2678e20ee0f9573d653c Parents: 97a2393 Author: root Authored: Wed Oct 18 21:49:58 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:50:14 2017 +0200 -- .../task_for_testing_log_view/2017-09-01T00.00.00/1.log| 1 + .../task_for_testing_log_view/2017-09-01T00:00:00/1.log| 1 - tests/www/test_views.py| 2 ++ 3 files changed, 3 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2da48568/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log -- diff --git a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log new file mode 100644 index 000..bc10ef7 --- /dev/null +++ b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log @@ -0,0 +1 @@ +Log for testing. http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2da48568/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log -- diff --git a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log deleted file mode 100644 index bc10ef7..000 --- a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log +++ /dev/null @@ -1 +0,0 @@ -Log for testing. http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2da48568/tests/www/test_views.py -- diff --git a/tests/www/test_views.py b/tests/www/test_views.py index 553e7cc..4931487 100644 --- a/tests/www/test_views.py +++ b/tests/www/test_views.py @@ -328,6 +328,8 @@ class TestLogView(unittest.TestCase): current_dir = os.path.dirname(os.path.abspath(__file__)) logging_config['handlers']['file.task']['base_log_folder'] = os.path.normpath( os.path.join(current_dir, 'test_logs')) +logging_config['handlers']['file.task']['filename_template'] = \ +'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts | replace(":", ".") }}/{{ try_number }}.log' # Write the custom logging configuration to a file self.settings_folder = tempfile.mkdtemp()
[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name
[ https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209888#comment-16209888 ] ASF subversion and git services commented on AIRFLOW-1722: -- Commit 97a2393f644ff438986843c04a47754a37ae0bc5 in incubator-airflow's branch refs/heads/master from [~mschmoyer] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=97a2393 ] [AIRFLOW-1722] Fix typo in scheduler autorestart output filename Extra r in /tmp/airflow_scheduler_errors.txt has been removed. Closes #2693 from mschmo/AIRFLOW-1722 > Typo in airflow_scheduler_autorestart.sh log file name > -- > > Key: AIRFLOW-1722 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1722 > Project: Apache Airflow > Issue Type: Task >Reporter: Matthew Schmoyer >Assignee: Matthew Schmoyer >Priority: Trivial > Original Estimate: 1m > Remaining Estimate: 1m > > Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name
[ https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209890#comment-16209890 ] ASF subversion and git services commented on AIRFLOW-1722: -- Commit fcc3c2f6e5860d5414752ace22160dbe19f68617 in incubator-airflow's branch refs/heads/v1-9-test from [~mschmoyer] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fcc3c2f ] [AIRFLOW-1722] Fix typo in scheduler autorestart output filename Extra r in /tmp/airflow_scheduler_errors.txt has been removed. Closes #2693 from mschmo/AIRFLOW-1722 (cherry picked from commit 97a2393f644ff438986843c04a47754a37ae0bc5) Signed-off-by: Bolke de Bruin > Typo in airflow_scheduler_autorestart.sh log file name > -- > > Key: AIRFLOW-1722 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1722 > Project: Apache Airflow > Issue Type: Task >Reporter: Matthew Schmoyer >Assignee: Matthew Schmoyer >Priority: Trivial > Original Estimate: 1m > Remaining Estimate: 1m > > Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name
[ https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209887#comment-16209887 ] ASF subversion and git services commented on AIRFLOW-1722: -- Commit 97a2393f644ff438986843c04a47754a37ae0bc5 in incubator-airflow's branch refs/heads/master from [~mschmoyer] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=97a2393 ] [AIRFLOW-1722] Fix typo in scheduler autorestart output filename Extra r in /tmp/airflow_scheduler_errors.txt has been removed. Closes #2693 from mschmo/AIRFLOW-1722 > Typo in airflow_scheduler_autorestart.sh log file name > -- > > Key: AIRFLOW-1722 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1722 > Project: Apache Airflow > Issue Type: Task >Reporter: Matthew Schmoyer >Assignee: Matthew Schmoyer >Priority: Trivial > Original Estimate: 1m > Remaining Estimate: 1m > > Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name
[ https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209889#comment-16209889 ] ASF subversion and git services commented on AIRFLOW-1722: -- Commit fcc3c2f6e5860d5414752ace22160dbe19f68617 in incubator-airflow's branch refs/heads/v1-9-test from [~mschmoyer] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fcc3c2f ] [AIRFLOW-1722] Fix typo in scheduler autorestart output filename Extra r in /tmp/airflow_scheduler_errors.txt has been removed. Closes #2693 from mschmo/AIRFLOW-1722 (cherry picked from commit 97a2393f644ff438986843c04a47754a37ae0bc5) Signed-off-by: Bolke de Bruin > Typo in airflow_scheduler_autorestart.sh log file name > -- > > Key: AIRFLOW-1722 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1722 > Project: Apache Airflow > Issue Type: Task >Reporter: Matthew Schmoyer >Assignee: Matthew Schmoyer >Priority: Trivial > Original Estimate: 1m > Remaining Estimate: 1m > > Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt` -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1722] Fix typo in scheduler autorestart output filename
Repository: incubator-airflow Updated Branches: refs/heads/v1-9-test ef4f98840 -> fcc3c2f6e [AIRFLOW-1722] Fix typo in scheduler autorestart output filename Extra r in /tmp/airflow_scheduler_errors.txt has been removed. Closes #2693 from mschmo/AIRFLOW-1722 (cherry picked from commit 97a2393f644ff438986843c04a47754a37ae0bc5) Signed-off-by: Bolke de Bruin Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fcc3c2f6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fcc3c2f6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fcc3c2f6 Branch: refs/heads/v1-9-test Commit: fcc3c2f6e5860d5414752ace22160dbe19f68617 Parents: ef4f988 Author: Matthew Schmoyer Authored: Wed Oct 18 21:41:01 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:41:17 2017 +0200 -- airflow/bin/airflow_scheduler_autorestart.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fcc3c2f6/airflow/bin/airflow_scheduler_autorestart.sh -- diff --git a/airflow/bin/airflow_scheduler_autorestart.sh b/airflow/bin/airflow_scheduler_autorestart.sh index a5f6fe7..910404d 100755 --- a/airflow/bin/airflow_scheduler_autorestart.sh +++ b/airflow/bin/airflow_scheduler_autorestart.sh @@ -14,6 +14,6 @@ while echo "Running"; do airflow scheduler -n 5 echo "Scheduler crashed with exit code $?. Respawning.." >&2 -date >> /tmp/airflow_scheduler_errrors.txt +date >> /tmp/airflow_scheduler_errors.txt sleep 1 done
incubator-airflow git commit: [AIRFLOW-1722] Fix typo in scheduler autorestart output filename
Repository: incubator-airflow Updated Branches: refs/heads/master 7cb818bba -> 97a2393f6 [AIRFLOW-1722] Fix typo in scheduler autorestart output filename Extra r in /tmp/airflow_scheduler_errors.txt has been removed. Closes #2693 from mschmo/AIRFLOW-1722 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/97a2393f Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/97a2393f Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/97a2393f Branch: refs/heads/master Commit: 97a2393f644ff438986843c04a47754a37ae0bc5 Parents: 7cb818b Author: Matthew Schmoyer Authored: Wed Oct 18 21:41:01 2017 +0200 Committer: Bolke de Bruin Committed: Wed Oct 18 21:41:01 2017 +0200 -- airflow/bin/airflow_scheduler_autorestart.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/97a2393f/airflow/bin/airflow_scheduler_autorestart.sh -- diff --git a/airflow/bin/airflow_scheduler_autorestart.sh b/airflow/bin/airflow_scheduler_autorestart.sh index a5f6fe7..910404d 100755 --- a/airflow/bin/airflow_scheduler_autorestart.sh +++ b/airflow/bin/airflow_scheduler_autorestart.sh @@ -14,6 +14,6 @@ while echo "Running"; do airflow scheduler -n 5 echo "Scheduler crashed with exit code $?. Respawning.." >&2 -date >> /tmp/airflow_scheduler_errrors.txt +date >> /tmp/airflow_scheduler_errors.txt sleep 1 done
[jira] [Resolved] (AIRFLOW-1723) Support sendgrid in email backend
[ https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini resolved AIRFLOW-1723. -- Resolution: Fixed Fix Version/s: 1.10.0 > Support sendgrid in email backend > -- > > Key: AIRFLOW-1723 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1723 > Project: Apache Airflow > Issue Type: New Feature > Components: utils >Affects Versions: 1.9.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > Fix For: 1.10.0 > > > Current airflow email backend only supports SMTP, this PR aims to extend > email backend and integrate with sendgrid. The airflow config file is also > updated to include a [sendgrid] section where user can specify api_key and > mail_from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1723) Support sendgrid in email backend
[ https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209874#comment-16209874 ] ASF subversion and git services commented on AIRFLOW-1723: -- Commit 7cb818bbacb2a2695282471591a9e323d8efbf5c in incubator-airflow's branch refs/heads/master from [~fenglu] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7cb818b ] [AIRFLOW-1723] Support sendgrid in email backend Closes #2695 from fenglu-g/master > Support sendgrid in email backend > -- > > Key: AIRFLOW-1723 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1723 > Project: Apache Airflow > Issue Type: New Feature > Components: utils >Affects Versions: 1.9.0 >Reporter: Feng Lu >Assignee: Feng Lu >Priority: Minor > Fix For: 1.10.0 > > > Current airflow email backend only supports SMTP, this PR aims to extend > email backend and integrate with sendgrid. The airflow config file is also > updated to include a [sendgrid] section where user can specify api_key and > mail_from. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1723] Support sendgrid in email backend
Repository: incubator-airflow Updated Branches: refs/heads/master 6078e753a -> 7cb818bba [AIRFLOW-1723] Support sendgrid in email backend Closes #2695 from fenglu-g/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7cb818bb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7cb818bb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7cb818bb Branch: refs/heads/master Commit: 7cb818bbacb2a2695282471591a9e323d8efbf5c Parents: 6078e75 Author: fenglu-g Authored: Wed Oct 18 12:27:14 2017 -0700 Committer: Chris Riccomini Committed: Wed Oct 18 12:27:14 2017 -0700 -- airflow/config_templates/default_airflow.cfg | 6 +++ airflow/utils/email.py | 43 +++ scripts/ci/requirements.txt | 1 + setup.py | 2 + tests/utils/test_email.py| 51 +++ 5 files changed, 103 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7cb818bb/airflow/config_templates/default_airflow.cfg -- diff --git a/airflow/config_templates/default_airflow.cfg b/airflow/config_templates/default_airflow.cfg index dee6dc7..fe20261 100644 --- a/airflow/config_templates/default_airflow.cfg +++ b/airflow/config_templates/default_airflow.cfg @@ -244,6 +244,12 @@ page_size = 100 email_backend = airflow.utils.email.send_email_smtp +[sendgrid] +# Recommend an API key with Mail.send permission only. +sendgrid_api_key = +sendgrid_mail_from = airf...@example.com + + [smtp] # If you want airflow to send emails on retries, failure, and you want to use # the airflow.utils.email.send_email_smtp function, you have to configure an http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7cb818bb/airflow/utils/email.py -- diff --git a/airflow/utils/email.py b/airflow/utils/email.py index fadd4d5..21ae707 100644 --- a/airflow/utils/email.py +++ b/airflow/utils/email.py @@ -21,13 +21,16 @@ from builtins import str from past.builtins import basestring import importlib +import mimetypes import os +import sendgrid import smtplib from email.mime.text import MIMEText from email.mime.multipart import MIMEMultipart from email.mime.application import MIMEApplication from email.utils import formatdate +from sendgrid.helpers.mail import Attachment, Content, Email, Mail, Personalization from airflow import configuration from airflow.exceptions import AirflowConfigException @@ -44,6 +47,46 @@ def send_email(to, subject, html_content, files=None, dryrun=False, cc=None, bcc return backend(to, subject, html_content, files=files, dryrun=dryrun, cc=cc, bcc=bcc, mime_subtype=mime_subtype) +def send_email_sendgrid(to, subject, html_content, files=None, dryrun=False, cc=None, bcc=None, mime_subtype='mixed'): +""" +Send an email with html content using sendgrid. +""" +mail = Mail() +mail.from_email = Email(configuration.get('sendgrid', 'SENDGRID_MAIL_FROM')) +mail.subject = subject + +# Add the list of to emails. +to = get_email_address_list(to) +personalization = Personalization() +for to_address in to: + personalization.add_to(Email(to_address)) +mail.add_personalization(personalization) +mail.add_content(Content('text/html', html_content)) + +# Add email attachment. +for fname in files or []: +basename = os.path.basename(fname) +attachment = Attachment() +with open(fname, "rb") as f: + attachment.content = base64.b64encode(f.read()) + attachment.type = mimetypes.guess_type(basename)[0] + attachment.filename = basename + attachment.disposition = "attachment" + attachment.content_id = '<%s>' % basename +mail.add_attachment(attachment) +_post_sendgrid_mail(mail.get()) + + +def _post_sendgrid_mail(mail_data): +log = LoggingMixin().log +sg = sendgrid.SendGridAPIClient(apikey=configuration.get('sendgrid', 'SENDGRID_API_KEY')) +response = sg.client.mail.send.post(request_body=mail_data) +# 2xx status code. +if response.status_code >= 200 and response.status_code < 300: +log.info('The following email with subject %s is successfully sent to sendgrid.' % subject) +else: +log.warning('Failed to send out email with subject %s, status code: %s' % (subject, response.status_code)) + def send_email_smtp(to, subject, html_content, files=None, dryrun=False, cc=None, bcc=None, mime_subtype='mixed'): """ Send an email with html content http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7cb81
[jira] [Resolved] (AIRFLOW-1506) Improve DataprocClusterCreateOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini resolved AIRFLOW-1506. -- Resolution: Duplicate > Improve DataprocClusterCreateOperator > - > > Key: AIRFLOW-1506 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1506 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, operators >Reporter: Yu Ishikawa >Assignee: Jessica Chen Fan > > h2. Goals > {{DataprocClusterCreateOperator}} should support {{$. > gceClusterConfig.serviceAccountScopes}} to specify scopes for a Dataproc > cluster. > For example, I guess some users would like to store the result to Google > Datastore and like this. In such a case, we have to put additional scopes to > access to the produces from the Dataproc cluster. > https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#gceclusterconfig -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1718) Increase num_retries polling value on Dataproc hook
[ https://issues.apache.org/jira/browse/AIRFLOW-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini resolved AIRFLOW-1718. -- Resolution: Fixed Fix Version/s: 1.10.0 > Increase num_retries polling value on Dataproc hook > --- > > Key: AIRFLOW-1718 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1718 > Project: Apache Airflow > Issue Type: Bug >Reporter: Crystal Qian >Assignee: Crystal Qian >Priority: Minor > Fix For: 1.10.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, `num_retries = 0` when execute() is called > (https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.HttpRequest-class.html#execute), > which causes intermittent 500 errors > (https://stackoverflow.com/questions/46522261/deadline-exceeded-when-airflow-runs-spark-jobs). > We should increase this to allow retries for internal Dataproc queries to > other services in the short-term; also seeing if the `num_retries` count can > be increased at the _google-api-python-client_ level in the long-term. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1718) Increase num_retries polling value on Dataproc hook
[ https://issues.apache.org/jira/browse/AIRFLOW-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209792#comment-16209792 ] ASF subversion and git services commented on AIRFLOW-1718: -- Commit 6078e753aac35aa4f5971a719d9f736c35396770 in incubator-airflow's branch refs/heads/master from [~cjqian] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6078e75 ] [AIRFLOW-1718] Set num_retries on Dataproc job request execution Closes #2696 from cjqian/1718 > Increase num_retries polling value on Dataproc hook > --- > > Key: AIRFLOW-1718 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1718 > Project: Apache Airflow > Issue Type: Bug >Reporter: Crystal Qian >Assignee: Crystal Qian >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > Currently, `num_retries = 0` when execute() is called > (https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.HttpRequest-class.html#execute), > which causes intermittent 500 errors > (https://stackoverflow.com/questions/46522261/deadline-exceeded-when-airflow-runs-spark-jobs). > We should increase this to allow retries for internal Dataproc queries to > other services in the short-term; also seeing if the `num_retries` count can > be increased at the _google-api-python-client_ level in the long-term. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1718] Set num_retries on Dataproc job request execution
Repository: incubator-airflow Updated Branches: refs/heads/master b7a1f8b9c -> 6078e753a [AIRFLOW-1718] Set num_retries on Dataproc job request execution Closes #2696 from cjqian/1718 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6078e753 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6078e753 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6078e753 Branch: refs/heads/master Commit: 6078e753aac35aa4f5971a719d9f736c35396770 Parents: b7a1f8b Author: Crystal Qian Authored: Wed Oct 18 11:30:07 2017 -0700 Committer: Chris Riccomini Committed: Wed Oct 18 11:30:11 2017 -0700 -- airflow/contrib/hooks/gcp_dataproc_hook.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6078e753/airflow/contrib/hooks/gcp_dataproc_hook.py -- diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py b/airflow/contrib/hooks/gcp_dataproc_hook.py index 4be166e..5b96484 100644 --- a/airflow/contrib/hooks/gcp_dataproc_hook.py +++ b/airflow/contrib/hooks/gcp_dataproc_hook.py @@ -41,7 +41,7 @@ class _DataProcJob(LoggingMixin): self.job = self.dataproc_api.projects().regions().jobs().get( projectId=self.project_id, region=self.region, -jobId=self.job_id).execute() +jobId=self.job_id).execute(num_retries=5) if 'ERROR' == self.job['status']['state']: print(str(self.job)) self.log.error('DataProc job %s has errors', self.job_id)
[jira] [Updated] (AIRFLOW-1727) Add unit tests for DataProcHook
[ https://issues.apache.org/jira/browse/AIRFLOW-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini updated AIRFLOW-1727: - Fix Version/s: 1.10.0 > Add unit tests for DataProcHook > --- > > Key: AIRFLOW-1727 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1727 > Project: Apache Airflow > Issue Type: Test >Reporter: Crystal Qian >Assignee: Crystal Qian >Priority: Minor > Fix For: 1.10.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (AIRFLOW-1727) Add unit tests for DataProcHook
[ https://issues.apache.org/jira/browse/AIRFLOW-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Riccomini resolved AIRFLOW-1727. -- Resolution: Fixed > Add unit tests for DataProcHook > --- > > Key: AIRFLOW-1727 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1727 > Project: Apache Airflow > Issue Type: Test >Reporter: Crystal Qian >Assignee: Crystal Qian >Priority: Minor > Fix For: 1.10.0 > > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1727) Add unit tests for DataProcHook
[ https://issues.apache.org/jira/browse/AIRFLOW-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209787#comment-16209787 ] ASF subversion and git services commented on AIRFLOW-1727: -- Commit b7a1f8b9cbbf795f663987254a681281cda46abf in incubator-airflow's branch refs/heads/master from [~cjqian] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b7a1f8b ] [AIRFLOW-1727] Add unit tests for DataProcHook Closes #2697 from cjqian/1727 > Add unit tests for DataProcHook > --- > > Key: AIRFLOW-1727 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1727 > Project: Apache Airflow > Issue Type: Test >Reporter: Crystal Qian >Assignee: Crystal Qian >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
incubator-airflow git commit: [AIRFLOW-1727] Add unit tests for DataProcHook
Repository: incubator-airflow Updated Branches: refs/heads/master cb868f49f -> b7a1f8b9c [AIRFLOW-1727] Add unit tests for DataProcHook Closes #2697 from cjqian/1727 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b7a1f8b9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b7a1f8b9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b7a1f8b9 Branch: refs/heads/master Commit: b7a1f8b9cbbf795f663987254a681281cda46abf Parents: cb868f4 Author: Crystal Qian Authored: Wed Oct 18 11:28:04 2017 -0700 Committer: Chris Riccomini Committed: Wed Oct 18 11:28:04 2017 -0700 -- tests/contrib/hooks/test_gcp_dataproc_hook.py | 48 ++ 1 file changed, 48 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b7a1f8b9/tests/contrib/hooks/test_gcp_dataproc_hook.py -- diff --git a/tests/contrib/hooks/test_gcp_dataproc_hook.py b/tests/contrib/hooks/test_gcp_dataproc_hook.py new file mode 100644 index 000..f49ec7d --- /dev/null +++ b/tests/contrib/hooks/test_gcp_dataproc_hook.py @@ -0,0 +1,48 @@ +# -*- coding: utf-8 -*- +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +import unittest +from airflow.contrib.hooks.gcp_dataproc_hook import DataProcHook + +try: +from unittest import mock +except ImportError: +try: +import mock +except ImportError: +mock = None + +JOB = 'test-job' +PROJECT_ID = 'test-project-id' +REGION = 'global' +TASK_ID = 'test-task-id' + +BASE_STRING = 'airflow.contrib.hooks.gcp_api_base_hook.{}' +DATAPROC_STRING = 'airflow.contrib.hooks.gcp_dataproc_hook.{}' + +def mock_init(self, gcp_conn_id, delegate_to=None): +pass + +class DataProcHookTest(unittest.TestCase): +def setUp(self): +with mock.patch(BASE_STRING.format('GoogleCloudBaseHook.__init__'), +new=mock_init): +self.dataproc_hook = DataProcHook() + +@mock.patch(DATAPROC_STRING.format('_DataProcJob')) +def test_submit(self, job_mock): + with mock.patch(DATAPROC_STRING.format('DataProcHook.get_conn', return_value=None)): +self.dataproc_hook.submit(PROJECT_ID, JOB) +job_mock.assert_called_once_with(mock.ANY, PROJECT_ID, JOB, REGION)
[jira] [Work started] (AIRFLOW-1728) Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on AIRFLOW-1728 started by Jessica Chen Fan . -- > Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator > > > Key: AIRFLOW-1728 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1728 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Jessica Chen Fan >Assignee: Jessica Chen Fan >Priority: Minor > > Add ability to specify subnetwork and firewall tags for easier networking > when creating > dataproc clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1506) Improve DataprocClusterCreateOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jessica Chen Fan reassigned AIRFLOW-1506: -- Assignee: Jessica Chen Fan > Improve DataprocClusterCreateOperator > - > > Key: AIRFLOW-1506 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1506 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, operators >Reporter: Yu Ishikawa >Assignee: Jessica Chen Fan > > h2. Goals > {{DataprocClusterCreateOperator}} should support {{$. > gceClusterConfig.serviceAccountScopes}} to specify scopes for a Dataproc > cluster. > For example, I guess some users would like to store the result to Google > Datastore and like this. In such a case, we have to put additional scopes to > access to the produces from the Dataproc cluster. > https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#gceclusterconfig -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-1506) Improve DataprocClusterCreateOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209501#comment-16209501 ] Jessica Chen Fan commented on AIRFLOW-1506: FYI this was resolved with [AIRFLOW-1535|https://issues.apache.org/jira/browse/AIRFLOW-1535]. > Improve DataprocClusterCreateOperator > - > > Key: AIRFLOW-1506 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1506 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, operators >Reporter: Yu Ishikawa > > h2. Goals > {{DataprocClusterCreateOperator}} should support {{$. > gceClusterConfig.serviceAccountScopes}} to specify scopes for a Dataproc > cluster. > For example, I guess some users would like to store the result to Google > Datastore and like this. In such a case, we have to put additional scopes to > access to the produces from the Dataproc cluster. > https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#gceclusterconfig -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (AIRFLOW-1728) Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator
[ https://issues.apache.org/jira/browse/AIRFLOW-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jessica Chen Fan updated AIRFLOW-1728: --- Summary: Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator (was: Add subnetworkUri and tags to DataprocClusterCreateOperator ) > Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator > > > Key: AIRFLOW-1728 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1728 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Jessica Chen Fan >Assignee: Jessica Chen Fan >Priority: Minor > > Add ability to specify subnetwork and firewall tags for easier networking > when creating > dataproc clusters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AIRFLOW-520) Show Airflow version on web page
[ https://issues.apache.org/jira/browse/AIRFLOW-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16209433#comment-16209433 ] Junyoung Park commented on AIRFLOW-520: --- You can see your version on the [About - versions] screen. > Show Airflow version on web page > > > Key: AIRFLOW-520 > URL: https://issues.apache.org/jira/browse/AIRFLOW-520 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Jakob Homan > > It would be nice if the currently running version were included somewhere on > the web page. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (AIRFLOW-1146) izip use in Python 3.4
[ https://issues.apache.org/jira/browse/AIRFLOW-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junyoung Park reassigned AIRFLOW-1146: -- Assignee: Junyoung Park > izip use in Python 3.4 > -- > > Key: AIRFLOW-1146 > URL: https://issues.apache.org/jira/browse/AIRFLOW-1146 > Project: Apache Airflow > Issue Type: Bug > Components: hive_hooks >Affects Versions: Airflow 1.8 >Reporter: Alexander Panzhin >Assignee: Junyoung Park > > Python 3 no longer has itertools.izip, but it is still used in > airflow/hooks/hive_hooks.py > This causes all kinds of havoc. > This needs fixed, if this is to be used on Python 3+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1731) Import custom config on PYTHONPATH
Fokko Driesprong created AIRFLOW-1731: - Summary: Import custom config on PYTHONPATH Key: AIRFLOW-1731 URL: https://issues.apache.org/jira/browse/AIRFLOW-1731 Project: Apache Airflow Issue Type: Bug Components: logging Affects Versions: 1.9.0 Reporter: Fokko Driesprong Fix For: 1.9.0 Currently the PYTHONPATH does not contain the required path to import a custom config as described. This needs to be fixed and the instructions needs to be updated based on user feedback. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1730) The value of XCom that queried from DB is not unpickled.
Shintaro Murakami created AIRFLOW-1730: -- Summary: The value of XCom that queried from DB is not unpickled. Key: AIRFLOW-1730 URL: https://issues.apache.org/jira/browse/AIRFLOW-1730 Project: Apache Airflow Issue Type: Bug Components: models, xcom Affects Versions: 1.9.0 Reporter: Shintaro Murakami Attachments: xcoms_by_example_xcom.png If enable_xcom_pickling is True, the value of XCom that queried from DB is not unpickled. The list of XComs not rendered correctly. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (AIRFLOW-1729) Ignore whole directories in .airflowignore
Cedric Hourcade created AIRFLOW-1729: Summary: Ignore whole directories in .airflowignore Key: AIRFLOW-1729 URL: https://issues.apache.org/jira/browse/AIRFLOW-1729 Project: Apache Airflow Issue Type: Improvement Components: core Affects Versions: Airflow 2.0 Reporter: Cedric Hourcade Priority: Minor The .airflowignore file allows to prevent scanning files for DAG. But even if we blacklist fulldirectory the {{os.walk}} will still go through them no matter how deep they are and skip files one by one, which can be an issue when you keep around big .git or virtualvenv directories. I suggest to add something like: {code} dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d)) for p in patterns])] {code} to prune the directories here: https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209 and in {{list_py_file_paths}} -- This message was sent by Atlassian JIRA (v6.4.14#64029)