[jira] [Created] (AIRFLOW-1736) Add HotelQuickly to Who Uses Airflow

[Jetsada Machom (JIRA)]

Jetsada Machom created AIRFLOW-1736:
---

 Summary: Add HotelQuickly to Who Uses Airflow
 Key: AIRFLOW-1736
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1736
 Project: Apache Airflow
  Issue Type: Task
Reporter: Jetsada Machom
Assignee: Jetsada Machom
Priority: Trivial






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1735) Log files do not show up for unscheduled dags

[Ace Haidrey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ace Haidrey updated AIRFLOW-1735:
-
Description: 
I created a new dag with a schedule of {{@once}} and left it unscheduled.  I 
ran different tasks a few times from the UI but the logs never showed up.  
Instead I'd see messages like the following:
{code}
*** Log file isn't local.
*** Fetching here: 
http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}

There is a log file for this task, but it seems to be at a slightly different 
path on the machine:

{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth 
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}

And indeed, when I change the URL to 
WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
 I see the actual log:

{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from 
~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava 
app IDs 
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}

I've added a screen shot of it below.

*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00 
and we can get that quite easily if it doesnt have it. You can see in this PR I 
made, but basically if we set
{code}
dttm = dateutil.parser.parse(request.args.get('execution_date'))
execution_date = dttm.isoformat()
{code}
This will fix it.

  was:
I created a new dag with a schedule of {{@once}} and left it unscheduled.  I 
ran different tasks a few times from the UI but the logs never showed up.  
Instead I'd see messages like the following:
{code}
*** Log file isn't local.
*** Fetching here: 
http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}

There is a log file for this task, but it seems to be at a slightly different 
path on the machine:

{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth 
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}

And indeed, when I change the URL to 
WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
 I see the actual log:

{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from 
~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava 
app IDs 
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}

I've added a screen shot of it below.

*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00 
and we can get that quite easily if it doesnt have it. You can see in this PR I 
made.


> Log files do not show up for unscheduled dags
> -
>
> Key: AIRFLOW-1735
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>  Labels: www
> Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png
>
>
> I created a new dag with a schedule of {{@once}} and left it unscheduled.  I 
> ran different tasks a few times from the UI but the logs never showed up.  
> Instead I'd see messages like the following:
> {code}
> *** Log file isn't local.
> *** Fetching here: 
> http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
> *** Failed to fetch log file from worker.
> {code}
> There is a log file for this task, but it seems to be at a slightly different 
> path on the machine:
> {code}
> (airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth 
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
> -rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 
> ./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
> {code}
> And indeed, when I change the URL to 
> 

[jira] [Created] (AIRFLOW-1735) Log files do not show up for unscheduled dags

[Ace Haidrey (JIRA)]

Ace Haidrey created AIRFLOW-1735:


 Summary: Log files do not show up for unscheduled dags
 Key: AIRFLOW-1735
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1735
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Ace Haidrey
Assignee: Ace Haidrey
 Attachments: Screen Shot 2017-06-21 at 10.53.28 PM.png

I created a new dag with a schedule of {{@once}} and left it unscheduled.  I 
ran different tasks a few times from the UI but the logs never showed up.  
Instead I'd see messages like the following:
{code}
*** Log file isn't local.
*** Fetching here: 
http://al-tools2.savagebeast.com:18700/log/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21
*** Failed to fetch log file from worker.
{code}

There is a log file for this task, but it seems to be at a slightly different 
path on the machine:

{code}
(airflow)hadoop@al-tools2:~/.virtualenvs/airflow$ ls -lth 
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00\:00\:00
-rw-rw-rw- 1 hadoop hadoop 5.2K Jun 21 21:57 
./airflow_home/logs/analytics_kochava_import_dag/kopandoraradioandroid1134523b395684da7_request/2017-06-21T00:00:00
{code}

And indeed, when I change the URL to 
WEB_SERVER/admin/airflow/log?task_id=TASK_ID_request_id=DAG_ID_date=2017-06-21T00:00:00,
 I see the actual log:

{code}
*** Reading local log.
[2017-06-21 21:57:20,931] {models.py:172} INFO - Filling up the DagBag from 
~/analytics_kochava_import_dag.py
[2017-06-21 21:57:21,307] {analytics_kochava_import_dag.py:139} INFO - Kochava 
app IDs 
...
[2017-06-21 21:57:26,400] {jobs.py:2083} INFO - Task exited with return code 0
{code}

I've added a screen shot of it below.

*FIX:*
I have noticed in www/view.py the issue is not have the time component 00:00:00 
and we can get that quite easily if it doesnt have it. You can see in this PR I 
made.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options

[Ace Haidrey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ace Haidrey updated AIRFLOW-1734:
-
Description: 
After taking a look at the Sqoop Operator I have found a number of errors with 
it. I will describe them in depth on the PR I will make, but it includes assign 
wrong parameters, not hiding the users raw password in the logs (though a 
method to do that had been created in the hook, wrong if conditions, and the 
inability to add other sqoop options that would make sense to have the option 
to pass those.

I have made some remarks in past commits about it too in the case I'm 
misinterpreting but regardless there are definitely errors.
https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796

Here is my PR with change by change explanation:
https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131

  was:
After taking a look at the Sqoop Operator I have found a number of errors with 
it. I will describe them in depth on the PR I will make, but it includes assign 
wrong parameters, not hiding the users raw password in the logs (though a 
method to do that had been created in the hook, wrong if conditions, and the 
inability to add other sqoop options that would make sense to have the option 
to pass those.

I have made some remarks in past commits about it too in the case I'm 
misinterpreting but regardless there are definitely errors.
https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796



> Sqoop Operator contains logic errors & needs options to pass more sqoop 
> options
> ---
>
> Key: AIRFLOW-1734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1734
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>  Labels: patch
>
> After taking a look at the Sqoop Operator I have found a number of errors 
> with it. I will describe them in depth on the PR I will make, but it includes 
> assign wrong parameters, not hiding the users raw password in the logs 
> (though a method to do that had been created in the hook, wrong if 
> conditions, and the inability to add other sqoop options that would make 
> sense to have the option to pass those.
> I have made some remarks in past commits about it too in the case I'm 
> misinterpreting but regardless there are definitely errors.
> https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796
> Here is my PR with change by change explanation:
> https://github.com/apache/incubator-airflow/pull/2703/files#diff-8e77f042c2e060bbfd60828431a91e9bL131



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options

[Ace Haidrey (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ace Haidrey updated AIRFLOW-1734:
-
Description: 
After taking a look at the Sqoop Operator I have found a number of errors with 
it. I will describe them in depth on the PR I will make, but it includes assign 
wrong parameters, not hiding the users raw password in the logs (though a 
method to do that had been created in the hook, wrong if conditions, and the 
inability to add other sqoop options that would make sense to have the option 
to pass those.

I have made some remarks in past commits about it too in the case I'm 
misinterpreting but regardless there are definitely errors.
https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796


  was:
After taking a look at the Sqoop Operator I have found a number of errors with 
it. I will describe them in depth on the PR I will make, but it includes 
passign wrong parameters, not hiding the users raw password in the logs (though 
a method to do that had been created in the hook, wrong if conditions, and the 
inability to add other sqoop options that would make sense to have the option 
to pass those.

I have made some remarks in past commits about it too in the case I'm 
misinterpreting but regardless there are definitely errors.
https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796



> Sqoop Operator contains logic errors & needs options to pass more sqoop 
> options
> ---
>
> Key: AIRFLOW-1734
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1734
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: contrib
>Reporter: Ace Haidrey
>Assignee: Ace Haidrey
>  Labels: patch
>
> After taking a look at the Sqoop Operator I have found a number of errors 
> with it. I will describe them in depth on the PR I will make, but it includes 
> assign wrong parameters, not hiding the users raw password in the logs 
> (though a method to do that had been created in the hook, wrong if 
> conditions, and the inability to add other sqoop options that would make 
> sense to have the option to pass those.
> I have made some remarks in past commits about it too in the case I'm 
> misinterpreting but regardless there are definitely errors.
> https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1734) Sqoop Operator contains logic errors & needs options to pass more sqoop options

[Ace Haidrey (JIRA)]

Ace Haidrey created AIRFLOW-1734:


 Summary: Sqoop Operator contains logic errors & needs options to 
pass more sqoop options
 Key: AIRFLOW-1734
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1734
 Project: Apache Airflow
  Issue Type: Bug
  Components: contrib
Reporter: Ace Haidrey
Assignee: Ace Haidrey


After taking a look at the Sqoop Operator I have found a number of errors with 
it. I will describe them in depth on the PR I will make, but it includes 
passign wrong parameters, not hiding the users raw password in the logs (though 
a method to do that had been created in the hook, wrong if conditions, and the 
inability to add other sqoop options that would make sense to have the option 
to pass those.

I have made some remarks in past commits about it too in the case I'm 
misinterpreting but regardless there are definitely errors.
https://github.com/apache/incubator-airflow/pull/2177#pullrequestreview-70046796




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1733) BashOperator readline can hang

[Dan Davydov (JIRA)]

Dan Davydov created AIRFLOW-1733:


 Summary: BashOperator readline can hang
 Key: AIRFLOW-1733
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1733
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Dan Davydov


If the child bash process dies for some reason, the BashOperator will hang 
since we use a blocking readline call to get output from the child. Instead we 
should read asynchronously or poll the child bash process like described in the 
solutions here:
https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1732) Improve Dataflow Hook Logging

[Trevor Edwards (JIRA)]

Trevor Edwards created AIRFLOW-1732:
---

 Summary: Improve Dataflow Hook Logging
 Key: AIRFLOW-1732
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1732
 Project: Apache Airflow
  Issue Type: Bug
Reporter: Trevor Edwards
Assignee: Trevor Edwards


Logging output for Dataflow hook could be a bit more useful. Namely:

# Log the command that is used for opening a dataflow subprocess
# If the dataflow subprocess experiences an error, log that error at warning 
(instead of debug)

Currently, errors are extremely opaque, only showing the exit code. The command 
used is unknown and the error is logged, but it is at the debug level which 
makes it difficult to find.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209921#comment-16209921
 ] 

ASF subversion and git services commented on AIRFLOW-1698:
--

Commit 00dd06eb0a387f85c1f16ce3aa17fc36473990bc in incubator-airflow's branch 
refs/heads/v1-9-test from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=00dd06e ]

[AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd

In the very early days, the Airflow scheduler
needed to be restarted
every so often to take new DAG_FOLDERS mutations
into account properly. This is no longer
required.

Closes #2677 from mistercrunch/scheduler_runs

(cherry picked from commit b464d23a6d9799b75ae53b3f858e779a48115fed)
Signed-off-by: Bolke de Bruin 


> Remove confusing SCHEDULER_RUNS env var from systemd scripts
> 
>
> Key: AIRFLOW-1698
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1698
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
> Fix For: 1.9.0
>
>
> In the very early days, the Airflow scheduler needed to be restarted
> every so often to take new DAG_FOLDERS mutations into account properly.
> This is no longer needed, so I'm removing artifacts in the
> systemd scripts.
> I'm unclear whether daemon scripts belong here. I can see how it's
> useful to have this somewhere,
> but it sits on the edge of what should be in the repo and can get
> out-of-sync.
> As a result it ends up creating more confusion than help.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1698.
-
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2677
[https://github.com/apache/incubator-airflow/pull/2677]

> Remove confusing SCHEDULER_RUNS env var from systemd scripts
> 
>
> Key: AIRFLOW-1698
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1698
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
> Fix For: 1.9.0
>
>
> In the very early days, the Airflow scheduler needed to be restarted
> every so often to take new DAG_FOLDERS mutations into account properly.
> This is no longer needed, so I'm removing artifacts in the
> systemd scripts.
> I'm unclear whether daemon scripts belong here. I can see how it's
> useful to have this somewhere,
> but it sits on the edge of what should be in the repo and can get
> out-of-sync.
> As a result it ends up creating more confusion than help.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1698) Remove confusing SCHEDULER_RUNS env var from systemd scripts

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209920#comment-16209920
 ] 

ASF subversion and git services commented on AIRFLOW-1698:
--

Commit b464d23a6d9799b75ae53b3f858e779a48115fed in incubator-airflow's branch 
refs/heads/master from [~maxime.beauche...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b464d23 ]

[AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd

In the very early days, the Airflow scheduler
needed to be restarted
every so often to take new DAG_FOLDERS mutations
into account properly. This is no longer
required.

Closes #2677 from mistercrunch/scheduler_runs


> Remove confusing SCHEDULER_RUNS env var from systemd scripts
> 
>
> Key: AIRFLOW-1698
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1698
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Maxime Beauchemin
> Fix For: 1.9.0
>
>
> In the very early days, the Airflow scheduler needed to be restarted
> every so often to take new DAG_FOLDERS mutations into account properly.
> This is no longer needed, so I'm removing artifacts in the
> systemd scripts.
> I'm unclear whether daemon scripts belong here. I can see how it's
> useful to have this somewhere,
> but it sits on the edge of what should be in the repo and can get
> out-of-sync.
> As a result it ends up creating more confusion than help.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 6110139a8 -> b464d23a6


[AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd

In the very early days, the Airflow scheduler
needed to be restarted
every so often to take new DAG_FOLDERS mutations
into account properly. This is no longer
required.

Closes #2677 from mistercrunch/scheduler_runs


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b464d23a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b464d23a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b464d23a

Branch: refs/heads/master
Commit: b464d23a6d9799b75ae53b3f858e779a48115fed
Parents: 6110139
Author: Maxime Beauchemin 
Authored: Wed Oct 18 21:55:54 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:55:57 2017 +0200

--
 scripts/systemd/airflow   | 4 +---
 scripts/systemd/airflow-scheduler.service | 4 ++--
 2 files changed, 3 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b464d23a/scripts/systemd/airflow
--
diff --git a/scripts/systemd/airflow b/scripts/systemd/airflow
index 5317548..bb2156b 100644
--- a/scripts/systemd/airflow
+++ b/scripts/systemd/airflow
@@ -16,6 +16,4 @@
 #
 # AIRFLOW_CONFIG=
 # AIRFLOW_HOME=
-#
-# required setting, 0 sets it to unlimited. Scheduler will get restart after 
every X runs
-SCHEDULER_RUNS=5
\ No newline at end of file
+

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b464d23a/scripts/systemd/airflow-scheduler.service
--
diff --git a/scripts/systemd/airflow-scheduler.service 
b/scripts/systemd/airflow-scheduler.service
index 95229d6..29daa39 100644
--- a/scripts/systemd/airflow-scheduler.service
+++ b/scripts/systemd/airflow-scheduler.service
@@ -21,9 +21,9 @@ EnvironmentFile=/etc/sysconfig/airflow
 User=airflow
 Group=airflow
 Type=simple
-ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS}
+ExecStart=/bin/airflow scheduler
 Restart=always
 RestartSec=5s
 
 [Install]
-WantedBy=multi-user.target
\ No newline at end of file
+WantedBy=multi-user.target

incubator-airflow git commit: [AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test c6e5ae7c5 -> 00dd06eb0


[AIRFLOW-1698] Remove SCHEDULER_RUNS env var in systemd

In the very early days, the Airflow scheduler
needed to be restarted
every so often to take new DAG_FOLDERS mutations
into account properly. This is no longer
required.

Closes #2677 from mistercrunch/scheduler_runs

(cherry picked from commit b464d23a6d9799b75ae53b3f858e779a48115fed)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/00dd06eb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/00dd06eb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/00dd06eb

Branch: refs/heads/v1-9-test
Commit: 00dd06eb0a387f85c1f16ce3aa17fc36473990bc
Parents: c6e5ae7
Author: Maxime Beauchemin 
Authored: Wed Oct 18 21:55:54 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:57:31 2017 +0200

--
 scripts/systemd/airflow   | 4 +---
 scripts/systemd/airflow-scheduler.service | 4 ++--
 2 files changed, 3 insertions(+), 5 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/00dd06eb/scripts/systemd/airflow
--
diff --git a/scripts/systemd/airflow b/scripts/systemd/airflow
index 5317548..bb2156b 100644
--- a/scripts/systemd/airflow
+++ b/scripts/systemd/airflow
@@ -16,6 +16,4 @@
 #
 # AIRFLOW_CONFIG=
 # AIRFLOW_HOME=
-#
-# required setting, 0 sets it to unlimited. Scheduler will get restart after 
every X runs
-SCHEDULER_RUNS=5
\ No newline at end of file
+

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/00dd06eb/scripts/systemd/airflow-scheduler.service
--
diff --git a/scripts/systemd/airflow-scheduler.service 
b/scripts/systemd/airflow-scheduler.service
index 95229d6..29daa39 100644
--- a/scripts/systemd/airflow-scheduler.service
+++ b/scripts/systemd/airflow-scheduler.service
@@ -21,9 +21,9 @@ EnvironmentFile=/etc/sysconfig/airflow
 User=airflow
 Group=airflow
 Type=simple
-ExecStart=/bin/airflow scheduler -n ${SCHEDULER_RUNS}
+ExecStart=/bin/airflow scheduler
 Restart=always
 RestartSec=5s
 
 [Install]
-WantedBy=multi-user.target
\ No newline at end of file
+WantedBy=multi-user.target

[jira] [Resolved] (AIRFLOW-1694) Hive Hooks: Python 3 does not have an `itertools.izip` function

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1694.
-
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2674
[https://github.com/apache/incubator-airflow/pull/2674]

> Hive Hooks: Python 3 does not have an `itertools.izip` function
> ---
>
> Key: AIRFLOW-1694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1694
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hive_hooks
>Reporter: Yati
> Fix For: 1.9.0
>
>
> In `hooks/hive_hooks.py`, we have a use of `itertools.izip()`, which would 
> not work in Python 3, but the builtin `zip()` returns an iterator. 
> `six.moves` can be used from either Python 2/3 to import such objects that 
> have been moved around.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1694] Stop using itertools.izip

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 2da485681 -> 6110139a8


[AIRFLOW-1694] Stop using itertools.izip

Itertools.zip does not exist in Python 3.

Closes #2674 from yati-sagade/fix-py3-zip


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6110139a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6110139a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6110139a

Branch: refs/heads/master
Commit: 6110139a8fd7069c87e3c99b3976865576180f67
Parents: 2da4856
Author: Yati Sagade 
Authored: Wed Oct 18 21:53:33 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:53:36 2017 +0200

--
 airflow/hooks/hive_hooks.py | 10 --
 1 file changed, 4 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6110139a/airflow/hooks/hive_hooks.py
--
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index 7c73491..a8f9c8f 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -14,7 +14,7 @@
 #
 
 from __future__ import print_function
-from builtins import zip
+from six.moves import zip
 from past.builtins import basestring
 
 import unicodecsv as csv
@@ -144,11 +144,9 @@ class HiveCliHook(BaseHook):
 if not d:
 return []
 return as_flattened_list(
-itertools.izip(
-["-hiveconf"] * len(d),
-["{}={}".format(k, v) for k, v in d.items()]
-)
-)
+zip(["-hiveconf"] * len(d),
+["{}={}".format(k, v) for k, v in d.items()])
+)
 
 def run_cli(self, hql, schema=None, verbose=True, hive_conf=None):
 """

incubator-airflow git commit: [AIRFLOW-1694] Stop using itertools.izip

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test 31805e836 -> c6e5ae7c5


[AIRFLOW-1694] Stop using itertools.izip

Itertools.zip does not exist in Python 3.

Closes #2674 from yati-sagade/fix-py3-zip

(cherry picked from commit 6110139a8fd7069c87e3c99b3976865576180f67)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c6e5ae7c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c6e5ae7c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c6e5ae7c

Branch: refs/heads/v1-9-test
Commit: c6e5ae7c57224ceb52cbf8d15b3362afd3616053
Parents: 31805e8
Author: Yati Sagade 
Authored: Wed Oct 18 21:53:33 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:54:24 2017 +0200

--
 airflow/hooks/hive_hooks.py | 10 --
 1 file changed, 4 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c6e5ae7c/airflow/hooks/hive_hooks.py
--
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index 7c73491..a8f9c8f 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -14,7 +14,7 @@
 #
 
 from __future__ import print_function
-from builtins import zip
+from six.moves import zip
 from past.builtins import basestring
 
 import unicodecsv as csv
@@ -144,11 +144,9 @@ class HiveCliHook(BaseHook):
 if not d:
 return []
 return as_flattened_list(
-itertools.izip(
-["-hiveconf"] * len(d),
-["{}={}".format(k, v) for k, v in d.items()]
-)
-)
+zip(["-hiveconf"] * len(d),
+["{}={}".format(k, v) for k, v in d.items()])
+)
 
 def run_cli(self, hql, schema=None, verbose=True, hive_conf=None):
 """

[jira] [Commented] (AIRFLOW-1694) Hive Hooks: Python 3 does not have an `itertools.izip` function

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209915#comment-16209915
 ] 

ASF subversion and git services commented on AIRFLOW-1694:
--

Commit 6110139a8fd7069c87e3c99b3976865576180f67 in incubator-airflow's branch 
refs/heads/master from Yati Sagade
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6110139 ]

[AIRFLOW-1694] Stop using itertools.izip

Itertools.zip does not exist in Python 3.

Closes #2674 from yati-sagade/fix-py3-zip


> Hive Hooks: Python 3 does not have an `itertools.izip` function
> ---
>
> Key: AIRFLOW-1694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1694
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hive_hooks
>Reporter: Yati
> Fix For: 1.9.0
>
>
> In `hooks/hive_hooks.py`, we have a use of `itertools.izip()`, which would 
> not work in Python 3, but the builtin `zip()` returns an iterator. 
> `six.moves` can be used from either Python 2/3 to import such objects that 
> have been moved around.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1694) Hive Hooks: Python 3 does not have an `itertools.izip` function

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209916#comment-16209916
 ] 

ASF subversion and git services commented on AIRFLOW-1694:
--

Commit c6e5ae7c57224ceb52cbf8d15b3362afd3616053 in incubator-airflow's branch 
refs/heads/v1-9-test from Yati Sagade
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=c6e5ae7 ]

[AIRFLOW-1694] Stop using itertools.izip

Itertools.zip does not exist in Python 3.

Closes #2674 from yati-sagade/fix-py3-zip

(cherry picked from commit 6110139a8fd7069c87e3c99b3976865576180f67)
Signed-off-by: Bolke de Bruin 


> Hive Hooks: Python 3 does not have an `itertools.izip` function
> ---
>
> Key: AIRFLOW-1694
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1694
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hive_hooks
>Reporter: Yati
> Fix For: 1.9.0
>
>
> In `hooks/hive_hooks.py`, we have a use of `itertools.izip()`, which would 
> not work in Python 3, but the builtin `zip()` returns an iterator. 
> `six.moves` can be used from either Python 2/3 to import such objects that 
> have been moved around.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1692) Master cannot be checked out on windows

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209907#comment-16209907
 ] 

ASF subversion and git services commented on AIRFLOW-1692:
--

Commit 31805e836a04700b5881705f58b33f32f2c8742a in incubator-airflow's branch 
refs/heads/v1-9-test from root
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=31805e8 ]

[AIRFLOW-1692] Change test_views filename to support Windows

Closes #2673 from NielsZeilemaker/AIRFLOW-1692

(cherry picked from commit 2da48568116f7382258e2678e20ee0f9573d653c)
Signed-off-by: Bolke de Bruin 


> Master cannot be checked out on windows
> ---
>
> Key: AIRFLOW-1692
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1692
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Niels Zeilemaker
>Assignee: Niels Zeilemaker
> Fix For: 1.9.0
>
>
> This file:
> /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
> Cannot be created on windows due to the colons in the directory name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1692) Master cannot be checked out on windows

[Bolke de Bruin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bolke de Bruin resolved AIRFLOW-1692.
-
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2673
[https://github.com/apache/incubator-airflow/pull/2673]

> Master cannot be checked out on windows
> ---
>
> Key: AIRFLOW-1692
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1692
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Niels Zeilemaker
>Assignee: Niels Zeilemaker
> Fix For: 1.9.0
>
>
> This file:
> /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
> Cannot be created on windows due to the colons in the directory name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1692) Master cannot be checked out on windows

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209906#comment-16209906
 ] 

ASF subversion and git services commented on AIRFLOW-1692:
--

Commit 31805e836a04700b5881705f58b33f32f2c8742a in incubator-airflow's branch 
refs/heads/v1-9-test from root
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=31805e8 ]

[AIRFLOW-1692] Change test_views filename to support Windows

Closes #2673 from NielsZeilemaker/AIRFLOW-1692

(cherry picked from commit 2da48568116f7382258e2678e20ee0f9573d653c)
Signed-off-by: Bolke de Bruin 


> Master cannot be checked out on windows
> ---
>
> Key: AIRFLOW-1692
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1692
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Niels Zeilemaker
>Assignee: Niels Zeilemaker
> Fix For: 1.9.0
>
>
> This file:
> /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
> Cannot be created on windows due to the colons in the directory name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1692) Master cannot be checked out on windows

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209905#comment-16209905
 ] 

ASF subversion and git services commented on AIRFLOW-1692:
--

Commit 2da48568116f7382258e2678e20ee0f9573d653c in incubator-airflow's branch 
refs/heads/master from root
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=2da4856 ]

[AIRFLOW-1692] Change test_views filename to support Windows

Closes #2673 from NielsZeilemaker/AIRFLOW-1692


> Master cannot be checked out on windows
> ---
>
> Key: AIRFLOW-1692
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1692
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Niels Zeilemaker
>Assignee: Niels Zeilemaker
>
> This file:
> /tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
> Cannot be created on windows due to the colons in the directory name



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1692] Change test_views filename to support Windows

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test fcc3c2f6e -> 31805e836


[AIRFLOW-1692] Change test_views filename to support Windows

Closes #2673 from NielsZeilemaker/AIRFLOW-1692

(cherry picked from commit 2da48568116f7382258e2678e20ee0f9573d653c)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/31805e83
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/31805e83
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/31805e83

Branch: refs/heads/v1-9-test
Commit: 31805e836a04700b5881705f58b33f32f2c8742a
Parents: fcc3c2f
Author: root 
Authored: Wed Oct 18 21:49:58 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:50:54 2017 +0200

--
 .../task_for_testing_log_view/2017-09-01T00.00.00/1.log| 1 +
 .../task_for_testing_log_view/2017-09-01T00:00:00/1.log| 1 -
 tests/www/test_views.py| 2 ++
 3 files changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/31805e83/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
--
diff --git 
a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
 
b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
new file mode 100644
index 000..bc10ef7
--- /dev/null
+++ 
b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
@@ -0,0 +1 @@
+Log for testing.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/31805e83/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
--
diff --git 
a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
 
b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
deleted file mode 100644
index bc10ef7..000
--- 
a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
+++ /dev/null
@@ -1 +0,0 @@
-Log for testing.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/31805e83/tests/www/test_views.py
--
diff --git a/tests/www/test_views.py b/tests/www/test_views.py
index 553e7cc..4931487 100644
--- a/tests/www/test_views.py
+++ b/tests/www/test_views.py
@@ -328,6 +328,8 @@ class TestLogView(unittest.TestCase):
 current_dir = os.path.dirname(os.path.abspath(__file__))
 logging_config['handlers']['file.task']['base_log_folder'] = 
os.path.normpath(
 os.path.join(current_dir, 'test_logs'))
+logging_config['handlers']['file.task']['filename_template'] = \
+'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts | replace(":", ".") }}/{{ 
try_number }}.log'
 
 # Write the custom logging configuration to a file
 self.settings_folder = tempfile.mkdtemp()

incubator-airflow git commit: [AIRFLOW-1692] Change test_views filename to support Windows

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 97a2393f6 -> 2da485681


[AIRFLOW-1692] Change test_views filename to support Windows

Closes #2673 from NielsZeilemaker/AIRFLOW-1692


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2da48568
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2da48568
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2da48568

Branch: refs/heads/master
Commit: 2da48568116f7382258e2678e20ee0f9573d653c
Parents: 97a2393
Author: root 
Authored: Wed Oct 18 21:49:58 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:50:14 2017 +0200

--
 .../task_for_testing_log_view/2017-09-01T00.00.00/1.log| 1 +
 .../task_for_testing_log_view/2017-09-01T00:00:00/1.log| 1 -
 tests/www/test_views.py| 2 ++
 3 files changed, 3 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2da48568/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
--
diff --git 
a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
 
b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
new file mode 100644
index 000..bc10ef7
--- /dev/null
+++ 
b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00.00.00/1.log
@@ -0,0 +1 @@
+Log for testing.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2da48568/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
--
diff --git 
a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
 
b/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
deleted file mode 100644
index bc10ef7..000
--- 
a/tests/www/test_logs/dag_for_testing_log_view/task_for_testing_log_view/2017-09-01T00:00:00/1.log
+++ /dev/null
@@ -1 +0,0 @@
-Log for testing.

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2da48568/tests/www/test_views.py
--
diff --git a/tests/www/test_views.py b/tests/www/test_views.py
index 553e7cc..4931487 100644
--- a/tests/www/test_views.py
+++ b/tests/www/test_views.py
@@ -328,6 +328,8 @@ class TestLogView(unittest.TestCase):
 current_dir = os.path.dirname(os.path.abspath(__file__))
 logging_config['handlers']['file.task']['base_log_folder'] = 
os.path.normpath(
 os.path.join(current_dir, 'test_logs'))
+logging_config['handlers']['file.task']['filename_template'] = \
+'{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts | replace(":", ".") }}/{{ 
try_number }}.log'
 
 # Write the custom logging configuration to a file
 self.settings_folder = tempfile.mkdtemp()

[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209888#comment-16209888
 ] 

ASF subversion and git services commented on AIRFLOW-1722:
--

Commit 97a2393f644ff438986843c04a47754a37ae0bc5 in incubator-airflow's branch 
refs/heads/master from [~mschmoyer]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=97a2393 ]

[AIRFLOW-1722] Fix typo in scheduler autorestart output filename

Extra r in /tmp/airflow_scheduler_errors.txt has
been removed.

Closes #2693 from mschmo/AIRFLOW-1722


> Typo in airflow_scheduler_autorestart.sh log file name
> --
>
> Key: AIRFLOW-1722
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1722
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Matthew Schmoyer
>Assignee: Matthew Schmoyer
>Priority: Trivial
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209889#comment-16209889
 ] 

ASF subversion and git services commented on AIRFLOW-1722:
--

Commit fcc3c2f6e5860d5414752ace22160dbe19f68617 in incubator-airflow's branch 
refs/heads/v1-9-test from [~mschmoyer]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fcc3c2f ]

[AIRFLOW-1722] Fix typo in scheduler autorestart output filename

Extra r in /tmp/airflow_scheduler_errors.txt has
been removed.

Closes #2693 from mschmo/AIRFLOW-1722

(cherry picked from commit 97a2393f644ff438986843c04a47754a37ae0bc5)
Signed-off-by: Bolke de Bruin 


> Typo in airflow_scheduler_autorestart.sh log file name
> --
>
> Key: AIRFLOW-1722
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1722
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Matthew Schmoyer
>Assignee: Matthew Schmoyer
>Priority: Trivial
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209890#comment-16209890
 ] 

ASF subversion and git services commented on AIRFLOW-1722:
--

Commit fcc3c2f6e5860d5414752ace22160dbe19f68617 in incubator-airflow's branch 
refs/heads/v1-9-test from [~mschmoyer]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fcc3c2f ]

[AIRFLOW-1722] Fix typo in scheduler autorestart output filename

Extra r in /tmp/airflow_scheduler_errors.txt has
been removed.

Closes #2693 from mschmo/AIRFLOW-1722

(cherry picked from commit 97a2393f644ff438986843c04a47754a37ae0bc5)
Signed-off-by: Bolke de Bruin 


> Typo in airflow_scheduler_autorestart.sh log file name
> --
>
> Key: AIRFLOW-1722
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1722
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Matthew Schmoyer
>Assignee: Matthew Schmoyer
>Priority: Trivial
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1722) Typo in airflow_scheduler_autorestart.sh log file name

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209887#comment-16209887
 ] 

ASF subversion and git services commented on AIRFLOW-1722:
--

Commit 97a2393f644ff438986843c04a47754a37ae0bc5 in incubator-airflow's branch 
refs/heads/master from [~mschmoyer]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=97a2393 ]

[AIRFLOW-1722] Fix typo in scheduler autorestart output filename

Extra r in /tmp/airflow_scheduler_errors.txt has
been removed.

Closes #2693 from mschmo/AIRFLOW-1722


> Typo in airflow_scheduler_autorestart.sh log file name
> --
>
> Key: AIRFLOW-1722
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1722
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Matthew Schmoyer
>Assignee: Matthew Schmoyer
>Priority: Trivial
>   Original Estimate: 1m
>  Remaining Estimate: 1m
>
> Rename `airflow_scheduler_errrors.txt` to `airflow_scheduler_errors.txt`



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1722] Fix typo in scheduler autorestart output filename

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/v1-9-test ef4f98840 -> fcc3c2f6e


[AIRFLOW-1722] Fix typo in scheduler autorestart output filename

Extra r in /tmp/airflow_scheduler_errors.txt has
been removed.

Closes #2693 from mschmo/AIRFLOW-1722

(cherry picked from commit 97a2393f644ff438986843c04a47754a37ae0bc5)
Signed-off-by: Bolke de Bruin 


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fcc3c2f6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fcc3c2f6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fcc3c2f6

Branch: refs/heads/v1-9-test
Commit: fcc3c2f6e5860d5414752ace22160dbe19f68617
Parents: ef4f988
Author: Matthew Schmoyer 
Authored: Wed Oct 18 21:41:01 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:41:17 2017 +0200

--
 airflow/bin/airflow_scheduler_autorestart.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fcc3c2f6/airflow/bin/airflow_scheduler_autorestart.sh
--
diff --git a/airflow/bin/airflow_scheduler_autorestart.sh 
b/airflow/bin/airflow_scheduler_autorestart.sh
index a5f6fe7..910404d 100755
--- a/airflow/bin/airflow_scheduler_autorestart.sh
+++ b/airflow/bin/airflow_scheduler_autorestart.sh
@@ -14,6 +14,6 @@
 while echo "Running"; do
 airflow scheduler -n 5
 echo "Scheduler crashed with exit code $?.  Respawning.." >&2
-date >> /tmp/airflow_scheduler_errrors.txt
+date >> /tmp/airflow_scheduler_errors.txt
 sleep 1
 done

incubator-airflow git commit: [AIRFLOW-1722] Fix typo in scheduler autorestart output filename

[bolke]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 7cb818bba -> 97a2393f6


[AIRFLOW-1722] Fix typo in scheduler autorestart output filename

Extra r in /tmp/airflow_scheduler_errors.txt has
been removed.

Closes #2693 from mschmo/AIRFLOW-1722


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/97a2393f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/97a2393f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/97a2393f

Branch: refs/heads/master
Commit: 97a2393f644ff438986843c04a47754a37ae0bc5
Parents: 7cb818b
Author: Matthew Schmoyer 
Authored: Wed Oct 18 21:41:01 2017 +0200
Committer: Bolke de Bruin 
Committed: Wed Oct 18 21:41:01 2017 +0200

--
 airflow/bin/airflow_scheduler_autorestart.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/97a2393f/airflow/bin/airflow_scheduler_autorestart.sh
--
diff --git a/airflow/bin/airflow_scheduler_autorestart.sh 
b/airflow/bin/airflow_scheduler_autorestart.sh
index a5f6fe7..910404d 100755
--- a/airflow/bin/airflow_scheduler_autorestart.sh
+++ b/airflow/bin/airflow_scheduler_autorestart.sh
@@ -14,6 +14,6 @@
 while echo "Running"; do
 airflow scheduler -n 5
 echo "Scheduler crashed with exit code $?.  Respawning.." >&2
-date >> /tmp/airflow_scheduler_errrors.txt
+date >> /tmp/airflow_scheduler_errors.txt
 sleep 1
 done

[jira] [Resolved] (AIRFLOW-1723) Support sendgrid in email backend

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini resolved AIRFLOW-1723.
--
   Resolution: Fixed
Fix Version/s: 1.10.0

> Support sendgrid in email backend 
> --
>
> Key: AIRFLOW-1723
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1723
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: utils
>Affects Versions: 1.9.0
>Reporter: Feng Lu
>Assignee: Feng Lu
>Priority: Minor
> Fix For: 1.10.0
>
>
> Current airflow email backend only supports SMTP, this PR aims to extend 
> email backend and integrate with sendgrid. The airflow config file is also 
> updated to include a [sendgrid] section where user can specify api_key and 
> mail_from. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1723) Support sendgrid in email backend

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209874#comment-16209874
 ] 

ASF subversion and git services commented on AIRFLOW-1723:
--

Commit 7cb818bbacb2a2695282471591a9e323d8efbf5c in incubator-airflow's branch 
refs/heads/master from [~fenglu]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7cb818b ]

[AIRFLOW-1723] Support sendgrid in email backend

Closes #2695 from fenglu-g/master


> Support sendgrid in email backend 
> --
>
> Key: AIRFLOW-1723
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1723
> Project: Apache Airflow
>  Issue Type: New Feature
>  Components: utils
>Affects Versions: 1.9.0
>Reporter: Feng Lu
>Assignee: Feng Lu
>Priority: Minor
> Fix For: 1.10.0
>
>
> Current airflow email backend only supports SMTP, this PR aims to extend 
> email backend and integrate with sendgrid. The airflow config file is also 
> updated to include a [sendgrid] section where user can specify api_key and 
> mail_from. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1723] Support sendgrid in email backend

[criccomini]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 6078e753a -> 7cb818bba


[AIRFLOW-1723] Support sendgrid in email backend

Closes #2695 from fenglu-g/master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7cb818bb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7cb818bb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7cb818bb

Branch: refs/heads/master
Commit: 7cb818bbacb2a2695282471591a9e323d8efbf5c
Parents: 6078e75
Author: fenglu-g 
Authored: Wed Oct 18 12:27:14 2017 -0700
Committer: Chris Riccomini 
Committed: Wed Oct 18 12:27:14 2017 -0700

--
 airflow/config_templates/default_airflow.cfg |  6 +++
 airflow/utils/email.py   | 43 +++
 scripts/ci/requirements.txt  |  1 +
 setup.py |  2 +
 tests/utils/test_email.py| 51 +++
 5 files changed, 103 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7cb818bb/airflow/config_templates/default_airflow.cfg
--
diff --git a/airflow/config_templates/default_airflow.cfg 
b/airflow/config_templates/default_airflow.cfg
index dee6dc7..fe20261 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -244,6 +244,12 @@ page_size = 100
 email_backend = airflow.utils.email.send_email_smtp
 
 
+[sendgrid]
+# Recommend an API key with Mail.send permission only.
+sendgrid_api_key = 
+sendgrid_mail_from = airf...@example.com
+
+
 [smtp]
 # If you want airflow to send emails on retries, failure, and you want to use
 # the airflow.utils.email.send_email_smtp function, you have to configure an

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7cb818bb/airflow/utils/email.py
--
diff --git a/airflow/utils/email.py b/airflow/utils/email.py
index fadd4d5..21ae707 100644
--- a/airflow/utils/email.py
+++ b/airflow/utils/email.py
@@ -21,13 +21,16 @@ from builtins import str
 from past.builtins import basestring
 
 import importlib
+import mimetypes
 import os
+import sendgrid
 import smtplib
 
 from email.mime.text import MIMEText
 from email.mime.multipart import MIMEMultipart
 from email.mime.application import MIMEApplication
 from email.utils import formatdate
+from sendgrid.helpers.mail import Attachment, Content, Email, Mail, 
Personalization
 
 from airflow import configuration
 from airflow.exceptions import AirflowConfigException
@@ -44,6 +47,46 @@ def send_email(to, subject, html_content, files=None, 
dryrun=False, cc=None, bcc
 return backend(to, subject, html_content, files=files, dryrun=dryrun, 
cc=cc, bcc=bcc, mime_subtype=mime_subtype)
 
 
+def send_email_sendgrid(to, subject, html_content, files=None, dryrun=False, 
cc=None, bcc=None, mime_subtype='mixed'):
+"""
+Send an email with html content using sendgrid.
+"""
+mail = Mail()
+mail.from_email = Email(configuration.get('sendgrid', 
'SENDGRID_MAIL_FROM'))
+mail.subject = subject
+
+# Add the list of to emails.
+to = get_email_address_list(to)
+personalization = Personalization()
+for to_address in to:
+  personalization.add_to(Email(to_address))
+mail.add_personalization(personalization)
+mail.add_content(Content('text/html', html_content))
+
+# Add email attachment.
+for fname in files or []:
+basename = os.path.basename(fname)
+attachment = Attachment()
+with open(fname, "rb") as f:
+  attachment.content = base64.b64encode(f.read())
+  attachment.type = mimetypes.guess_type(basename)[0]
+  attachment.filename = basename
+  attachment.disposition = "attachment"
+  attachment.content_id = '<%s>' % basename
+mail.add_attachment(attachment)
+_post_sendgrid_mail(mail.get())
+
+
+def _post_sendgrid_mail(mail_data):
+log = LoggingMixin().log
+sg = sendgrid.SendGridAPIClient(apikey=configuration.get('sendgrid', 
'SENDGRID_API_KEY'))
+response = sg.client.mail.send.post(request_body=mail_data)
+# 2xx status code.
+if response.status_code >= 200 and response.status_code < 300:
+log.info('The following email with subject %s is successfully sent to 
sendgrid.' % subject)
+else:
+log.warning('Failed to send out email with subject %s, status code: 
%s' % (subject, response.status_code))
+
 def send_email_smtp(to, subject, html_content, files=None, dryrun=False, 
cc=None, bcc=None, mime_subtype='mixed'):
 """
 Send an email with html content

[jira] [Resolved] (AIRFLOW-1506) Improve DataprocClusterCreateOperator

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini resolved AIRFLOW-1506.
--
Resolution: Duplicate

> Improve DataprocClusterCreateOperator
> -
>
> Key: AIRFLOW-1506
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1506
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, operators
>Reporter: Yu Ishikawa
>Assignee: Jessica Chen Fan 
>
> h2. Goals
> {{DataprocClusterCreateOperator}} should support {{$. 
> gceClusterConfig.serviceAccountScopes}} to specify scopes for a Dataproc 
> cluster.
> For example, I guess some users would like to store the result to Google 
> Datastore and like this. In such a case, we have to put additional scopes to 
> access to the produces from the Dataproc cluster.
> https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#gceclusterconfig



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1718) Increase num_retries polling value on Dataproc hook

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini resolved AIRFLOW-1718.
--
   Resolution: Fixed
Fix Version/s: 1.10.0

> Increase num_retries polling value on Dataproc hook
> ---
>
> Key: AIRFLOW-1718
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1718
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Crystal Qian
>Assignee: Crystal Qian
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, `num_retries = 0` when execute() is called 
> (https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.HttpRequest-class.html#execute),
>  which causes intermittent 500 errors 
> (https://stackoverflow.com/questions/46522261/deadline-exceeded-when-airflow-runs-spark-jobs).
>  We should increase this to allow retries for internal Dataproc queries to 
> other services in the short-term; also seeing if the `num_retries` count can 
> be increased at the _google-api-python-client_ level in the long-term.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1718) Increase num_retries polling value on Dataproc hook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209792#comment-16209792
 ] 

ASF subversion and git services commented on AIRFLOW-1718:
--

Commit 6078e753aac35aa4f5971a719d9f736c35396770 in incubator-airflow's branch 
refs/heads/master from [~cjqian]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6078e75 ]

[AIRFLOW-1718] Set num_retries on Dataproc job request execution

Closes #2696 from cjqian/1718


> Increase num_retries polling value on Dataproc hook
> ---
>
> Key: AIRFLOW-1718
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1718
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Crystal Qian
>Assignee: Crystal Qian
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently, `num_retries = 0` when execute() is called 
> (https://google.github.io/google-api-python-client/docs/epy/googleapiclient.http.HttpRequest-class.html#execute),
>  which causes intermittent 500 errors 
> (https://stackoverflow.com/questions/46522261/deadline-exceeded-when-airflow-runs-spark-jobs).
>  We should increase this to allow retries for internal Dataproc queries to 
> other services in the short-term; also seeing if the `num_retries` count can 
> be increased at the _google-api-python-client_ level in the long-term.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

incubator-airflow git commit: [AIRFLOW-1718] Set num_retries on Dataproc job request execution

[criccomini]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master b7a1f8b9c -> 6078e753a


[AIRFLOW-1718] Set num_retries on Dataproc job request execution

Closes #2696 from cjqian/1718


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6078e753
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6078e753
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6078e753

Branch: refs/heads/master
Commit: 6078e753aac35aa4f5971a719d9f736c35396770
Parents: b7a1f8b
Author: Crystal Qian 
Authored: Wed Oct 18 11:30:07 2017 -0700
Committer: Chris Riccomini 
Committed: Wed Oct 18 11:30:11 2017 -0700

--
 airflow/contrib/hooks/gcp_dataproc_hook.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6078e753/airflow/contrib/hooks/gcp_dataproc_hook.py
--
diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py 
b/airflow/contrib/hooks/gcp_dataproc_hook.py
index 4be166e..5b96484 100644
--- a/airflow/contrib/hooks/gcp_dataproc_hook.py
+++ b/airflow/contrib/hooks/gcp_dataproc_hook.py
@@ -41,7 +41,7 @@ class _DataProcJob(LoggingMixin):
 self.job = self.dataproc_api.projects().regions().jobs().get(
 projectId=self.project_id,
 region=self.region,
-jobId=self.job_id).execute()
+jobId=self.job_id).execute(num_retries=5)
 if 'ERROR' == self.job['status']['state']:
 print(str(self.job))
 self.log.error('DataProc job %s has errors', self.job_id)

[jira] [Updated] (AIRFLOW-1727) Add unit tests for DataProcHook

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini updated AIRFLOW-1727:
-
Fix Version/s: 1.10.0

> Add unit tests for DataProcHook
> ---
>
> Key: AIRFLOW-1727
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1727
> Project: Apache Airflow
>  Issue Type: Test
>Reporter: Crystal Qian
>Assignee: Crystal Qian
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (AIRFLOW-1727) Add unit tests for DataProcHook

[Chris Riccomini (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Riccomini resolved AIRFLOW-1727.
--
Resolution: Fixed

> Add unit tests for DataProcHook
> ---
>
> Key: AIRFLOW-1727
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1727
> Project: Apache Airflow
>  Issue Type: Test
>Reporter: Crystal Qian
>Assignee: Crystal Qian
>Priority: Minor
> Fix For: 1.10.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1727) Add unit tests for DataProcHook

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209787#comment-16209787
 ] 

ASF subversion and git services commented on AIRFLOW-1727:
--

Commit b7a1f8b9cbbf795f663987254a681281cda46abf in incubator-airflow's branch 
refs/heads/master from [~cjqian]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=b7a1f8b ]

[AIRFLOW-1727] Add unit tests for DataProcHook

Closes #2697 from cjqian/1727


> Add unit tests for DataProcHook
> ---
>
> Key: AIRFLOW-1727
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1727
> Project: Apache Airflow
>  Issue Type: Test
>Reporter: Crystal Qian
>Assignee: Crystal Qian
>Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (AIRFLOW-1728) Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator

[Jessica Chen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on AIRFLOW-1728 started by Jessica Chen Fan .
--
> Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator 
> 
>
> Key: AIRFLOW-1728
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1728
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Jessica Chen Fan 
>Assignee: Jessica Chen Fan 
>Priority: Minor
>
> Add ability to specify subnetwork and firewall tags for easier networking 
> when creating 
> dataproc clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (AIRFLOW-1506) Improve DataprocClusterCreateOperator

[Jessica Chen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jessica Chen Fan  reassigned AIRFLOW-1506:
--

Assignee: Jessica Chen Fan 

> Improve DataprocClusterCreateOperator
> -
>
> Key: AIRFLOW-1506
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1506
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, operators
>Reporter: Yu Ishikawa
>Assignee: Jessica Chen Fan 
>
> h2. Goals
> {{DataprocClusterCreateOperator}} should support {{$. 
> gceClusterConfig.serviceAccountScopes}} to specify scopes for a Dataproc 
> cluster.
> For example, I guess some users would like to store the result to Google 
> Datastore and like this. In such a case, we have to put additional scopes to 
> access to the produces from the Dataproc cluster.
> https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#gceclusterconfig



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-1506) Improve DataprocClusterCreateOperator

[Jessica Chen Fan (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209501#comment-16209501
 ] 

Jessica Chen Fan  commented on AIRFLOW-1506:


FYI this was resolved with 
[AIRFLOW-1535|https://issues.apache.org/jira/browse/AIRFLOW-1535]. 

> Improve DataprocClusterCreateOperator
> -
>
> Key: AIRFLOW-1506
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1506
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: contrib, operators
>Reporter: Yu Ishikawa
>
> h2. Goals
> {{DataprocClusterCreateOperator}} should support {{$. 
> gceClusterConfig.serviceAccountScopes}} to specify scopes for a Dataproc 
> cluster.
> For example, I guess some users would like to store the result to Google 
> Datastore and like this. In such a case, we have to put additional scopes to 
> access to the produces from the Dataproc cluster.
> https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.clusters#gceclusterconfig



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (AIRFLOW-1728) Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator

[Jessica Chen Fan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jessica Chen Fan  updated AIRFLOW-1728:
---
Summary: Add networkUri, subnetworkUri and tags to 
DataprocClusterCreateOperator   (was: Add subnetworkUri and tags to 
DataprocClusterCreateOperator )

> Add networkUri, subnetworkUri and tags to DataprocClusterCreateOperator 
> 
>
> Key: AIRFLOW-1728
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1728
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Jessica Chen Fan 
>Assignee: Jessica Chen Fan 
>Priority: Minor
>
> Add ability to specify subnetwork and firewall tags for easier networking 
> when creating 
> dataproc clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (AIRFLOW-520) Show Airflow version on web page

[Junyoung Park (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209433#comment-16209433
 ] 

Junyoung Park commented on AIRFLOW-520:
---

You can see your version on the [About - versions] screen.

> Show Airflow version on web page
> 
>
> Key: AIRFLOW-520
> URL: https://issues.apache.org/jira/browse/AIRFLOW-520
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Jakob Homan
>
> It would be nice if the currently running version were included somewhere on 
> the web page.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (AIRFLOW-1146) izip use in Python 3.4

[Junyoung Park (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junyoung Park reassigned AIRFLOW-1146:
--

Assignee: Junyoung Park

> izip use in Python 3.4
> --
>
> Key: AIRFLOW-1146
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1146
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: hive_hooks
>Affects Versions: Airflow 1.8
>Reporter: Alexander Panzhin
>Assignee: Junyoung Park
>
> Python 3 no longer has itertools.izip, but it is still used in 
> airflow/hooks/hive_hooks.py
> This causes all kinds of havoc.
> This needs fixed, if this is to be used on Python 3+



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1731) Import custom config on PYTHONPATH

[Fokko Driesprong (JIRA)]

Fokko Driesprong created AIRFLOW-1731:
-

 Summary: Import custom config on PYTHONPATH
 Key: AIRFLOW-1731
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1731
 Project: Apache Airflow
  Issue Type: Bug
  Components: logging
Affects Versions: 1.9.0
Reporter: Fokko Driesprong
 Fix For: 1.9.0


Currently the PYTHONPATH does not contain the required path to import a custom 
config as described. This needs to be fixed and the instructions needs to be 
updated based on user feedback.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1730) The value of XCom that queried from DB is not unpickled.

[Shintaro Murakami (JIRA)]

Shintaro Murakami created AIRFLOW-1730:
--

 Summary: The value of XCom that queried from DB is not unpickled.
 Key: AIRFLOW-1730
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1730
 Project: Apache Airflow
  Issue Type: Bug
  Components: models, xcom
Affects Versions: 1.9.0
Reporter: Shintaro Murakami
 Attachments: xcoms_by_example_xcom.png

If enable_xcom_pickling is True, the value of XCom that queried from DB is not 
unpickled.
The list of XComs not rendered correctly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (AIRFLOW-1729) Ignore whole directories in .airflowignore

[Cedric Hourcade (JIRA)]

Cedric Hourcade created AIRFLOW-1729:


 Summary: Ignore whole directories in .airflowignore
 Key: AIRFLOW-1729
 URL: https://issues.apache.org/jira/browse/AIRFLOW-1729
 Project: Apache Airflow
  Issue Type: Improvement
  Components: core
Affects Versions: Airflow 2.0
Reporter: Cedric Hourcade
Priority: Minor


The .airflowignore file allows to prevent scanning files for DAG. But even if 
we blacklist fulldirectory the {{os.walk}} will still go through them no matter 
how deep they are and skip files one by one, which can be an issue when you 
keep around big .git or virtualvenv directories.

I suggest to add something like:
{code}
dirs[:] = [d for d in dirs if not any([re.findall(p, os.path.join(root, d)) for 
p in patterns])]
{code}
to prune the directories here: 
https://github.com/apache/incubator-airflow/blob/cfc2f73c445074e1e09d6ef6a056cd2b33a945da/airflow/utils/dag_processing.py#L208-L209
 and in {{list_py_file_paths}}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)