svn commit: r18770 - /dev/incubator/airflow/

[bolke]

Author: bolke
Date: Thu Mar 16 23:58:10 2017
New Revision: 18770

Log:
Removed old candidates

Added:
dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz   (with 
props)
dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.sha
Removed:
dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0b3+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0b3+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0b3+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0b4+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0b4+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0b4+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0b5+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0b5+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0b5+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0b5+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0rc1+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0rc1+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0rc1+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0rc1+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0rc2+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0rc2+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0rc2+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0rc2+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0rc3+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0rc3+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0rc3+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0rc3+apache.incubating.tar.gz.sha
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.asc
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.md5
dev/incubator/airflow/airflow-1.8.0rc4+apache.incubating.tar.gz.sha

Added: dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz
==
Binary file - no diff available.

Propchange: dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz
--
svn:mime-type = application/octet-stream

Added: dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.asc
==
--- dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.asc (added)
+++ dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.asc Thu Mar 
16 23:58:10 2017
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIcBAABCgAGBQJYxiRgAAoJEDUZC4PZBaC6m5wQALO1B3FbyaY/m18iaDDCDhCu
+Fs9By13vkpgpEl6cnwzjYsmJp+A27DVg7AtXw3xu5XDXU02fQaoMm1ISaBSJS4lG
+3jdEClHm96aQvWTHAIiyQGnoWqjmVQkTBHYa8gEZ6ZGl3p1RWOBv/n8QKeoTX/+j
+GgAETQxb5CSQZvR0pFjP5njPEAApbEC6L/y+8JbPhKF0EXA30jmTMMllQ8EiCIpc
+u0CfRuuCsuqQDzR9erw2pPSwqnUVZhipQhbsqUGlJcjmzUcubSEBUplMMOLdrJfX
+Idx/Beq3LMb76Qc/z+R4h0ScfCZpPn5/b1Zm58C1CaEbC+oridC8RQbZg19gN1Mw
+vOMilp4C3fkegoddOD6RttnUKCDam+8B0Q2SyYBfw15RjOJUtJXEzrUxgXxK5zCU
+czkWINmKtL0dJr2kaybFkUCkXmLZyjet36Y50lRWDqxkBbBtjpz8BrKqVqOXPQCA
+URMqA2/Mh+F8kLFk2yew2meiH0Pa8MS4r2fkwNzSCfMFCpVCrUeVyBef6Muh4ma8
+shjYs760yGgllIn+2LSdslvvkNoTbu4lAAhILKNTaArzN7MHro0V6yAsjLsGT4Sb
+rXe13fnaPILtnhwaT9ZR2vbWLLaS1HLTI9CnVs+X31t7XV2Pfy9JbAzbKOX3kK9O
+5eDlk7evJ+dv5UapXY9L
+=vGDz
+-END PGP SIGNATURE-

Added: dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.md5
==
--- dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.md5 (added)
+++ dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.md5 Thu Mar 
16 23:58:10 2017
@@ -0,0 +1,2 @@
+airflow-1.8.0rc5+apache.incubating.tar.gz: 
+60 2F BB 82 6E CD 2B 2D  37 65 6F A8 8E D8 82 33

Added: dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.sha
==
--- dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.sha (added)
+++ dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.sha Thu Mar 
16 23:58:10 2017
@@ 

[jira] [Reopened] (AIRFLOW-995) Update Github PR template

[Jeremiah Lowin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Lowin reopened AIRFLOW-995:


Reopening because the new template has "AIRFLOW-1" in it (as an example) which 
is picked up by the PR tool and inserted into the commit subjects

> Update Github PR template
> -
>
> Key: AIRFLOW-995
> URL: https://issues.apache.org/jira/browse/AIRFLOW-995
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>Priority: Minor
> Fix For: 1.9.0
>
>
> The Github PR template looks great when rendered but is harder to parse while 
> in editing mode (which is how all PR authors initially see it). A new 
> template would be clear whether editing or previewing and include checkboxes 
> for force some user acknowledgement of each required step.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (AIRFLOW-705) airflow.configuration.run_command output does not decode command output

[Li Xuanji (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Xuanji resolved AIRFLOW-705.
---
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2053
[https://github.com/apache/incubator-airflow/pull/2053]

> airflow.configuration.run_command output does not decode command output
> ---
>
> Key: AIRFLOW-705
> URL: https://issues.apache.org/jira/browse/AIRFLOW-705
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1.3
>Reporter: Scott Kruger
> Fix For: 1.9.0
>
>
> In python3, command outputs (to {{stdout}} and {{stderr}}) are of type 
> {{bytes}}, not {{str}}.  This causes the following exception when the 
> {{sql_alchemy_conn_cmd}} option is used:
> {noformat}
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 4, in 
> from airflow import configuration
>   File "/usr/local/lib/python3.5/site-packages/airflow/__init__.py", line 29, 
> in 
> from airflow import configuration as conf
>   File "/usr/local/lib/python3.5/site-packages/airflow/configuration.py", 
> line 657, in 
> conf.read(AIRFLOW_CONFIG)
>   File "/usr/local/lib/python3.5/site-packages/airflow/configuration.py", 
> line 519, in read
> self._validate()
>   File "/usr/local/lib/python3.5/site-packages/airflow/configuration.py", 
> line 447, in _validate
> "sqlite" in self.get('core', 'sql_alchemy_conn')):
> TypeError: a bytes-like object is required, not 'str'
> {noformat}
> For compatibility between python 2 and 3, 
> {{airflow.configuration.run_command}} should decode {{stdout}} to produce a 
> (unicode) string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-706) Configuration shell commands are not split properly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929175#comment-15929175
 ] 

ASF subversion and git services commented on AIRFLOW-706:
-

Commit 0bb6f2f458ddf305d77fc63d5e817c9b6eb22270 in incubator-airflow's branch 
refs/heads/master from [~thesquelched]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0bb6f2f ]

[AIRFLOW-705][AIRFLOW-706] Fix run_command bugs

`run_command` currently uses `str.split` for shell
commands, but this
breaks commands like `echo "foo bar"`.  Instead,
use `shlex.split`,
which understands how to handle quotes.

In python3, subprocess functions return
stdout/stderr as `bytes`, not
`str`.  To fix py2/py3 compatibility,
`run_command` now decodes the
output so that it always returns a `str`.

Dear Airflow Maintainers,

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-705
-
https://issues.apache.org/jira/browse/AIRFLOW-706

Testing Done:
- Changed `sql_alchemy_conn` to
`sql_alchemy_conn_cmd` in test configuration to
exercise `run_command` in tests

Closes #2053 from thesquelched/AIRFLOW-706


> Configuration shell commands are not split properly
> ---
>
> Key: AIRFLOW-706
> URL: https://issues.apache.org/jira/browse/AIRFLOW-706
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1.3
>Reporter: Scott Kruger
> Fix For: 1.9.0
>
>
> The command string in {{airflow.configuration.run_command}} is split with 
> {{str.split}} instead of {{shlex.split}}; this can cause problems when a 
> valid command includes an argument with spaces, e.g.
> {noformat}
> echo "foo bar"
> {noformat}
>  is split into 
> {noformat}
> ['echo', '"foo', 'bar"']
> {noformat}
> See: 
> https://github.com/apache/incubator-airflow/blob/master/airflow/configuration.py#L79



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-706) Configuration shell commands are not split properly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929176#comment-15929176
 ] 

ASF subversion and git services commented on AIRFLOW-706:
-

Commit 0bb6f2f458ddf305d77fc63d5e817c9b6eb22270 in incubator-airflow's branch 
refs/heads/master from [~thesquelched]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0bb6f2f ]

[AIRFLOW-705][AIRFLOW-706] Fix run_command bugs

`run_command` currently uses `str.split` for shell
commands, but this
breaks commands like `echo "foo bar"`.  Instead,
use `shlex.split`,
which understands how to handle quotes.

In python3, subprocess functions return
stdout/stderr as `bytes`, not
`str`.  To fix py2/py3 compatibility,
`run_command` now decodes the
output so that it always returns a `str`.

Dear Airflow Maintainers,

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-705
-
https://issues.apache.org/jira/browse/AIRFLOW-706

Testing Done:
- Changed `sql_alchemy_conn` to
`sql_alchemy_conn_cmd` in test configuration to
exercise `run_command` in tests

Closes #2053 from thesquelched/AIRFLOW-706


> Configuration shell commands are not split properly
> ---
>
> Key: AIRFLOW-706
> URL: https://issues.apache.org/jira/browse/AIRFLOW-706
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1.3
>Reporter: Scott Kruger
> Fix For: 1.9.0
>
>
> The command string in {{airflow.configuration.run_command}} is split with 
> {{str.split}} instead of {{shlex.split}}; this can cause problems when a 
> valid command includes an argument with spaces, e.g.
> {noformat}
> echo "foo bar"
> {noformat}
>  is split into 
> {noformat}
> ['echo', '"foo', 'bar"']
> {noformat}
> See: 
> https://github.com/apache/incubator-airflow/blob/master/airflow/configuration.py#L79



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-706) Configuration shell commands are not split properly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929173#comment-15929173
 ] 

ASF subversion and git services commented on AIRFLOW-706:
-

Commit 0bb6f2f458ddf305d77fc63d5e817c9b6eb22270 in incubator-airflow's branch 
refs/heads/master from [~thesquelched]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0bb6f2f ]

[AIRFLOW-705][AIRFLOW-706] Fix run_command bugs

`run_command` currently uses `str.split` for shell
commands, but this
breaks commands like `echo "foo bar"`.  Instead,
use `shlex.split`,
which understands how to handle quotes.

In python3, subprocess functions return
stdout/stderr as `bytes`, not
`str`.  To fix py2/py3 compatibility,
`run_command` now decodes the
output so that it always returns a `str`.

Dear Airflow Maintainers,

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-705
-
https://issues.apache.org/jira/browse/AIRFLOW-706

Testing Done:
- Changed `sql_alchemy_conn` to
`sql_alchemy_conn_cmd` in test configuration to
exercise `run_command` in tests

Closes #2053 from thesquelched/AIRFLOW-706


> Configuration shell commands are not split properly
> ---
>
> Key: AIRFLOW-706
> URL: https://issues.apache.org/jira/browse/AIRFLOW-706
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: Airflow 1.7.1.3
>Reporter: Scott Kruger
> Fix For: 1.9.0
>
>
> The command string in {{airflow.configuration.run_command}} is split with 
> {{str.split}} instead of {{shlex.split}}; this can cause problems when a 
> valid command includes an argument with spaces, e.g.
> {noformat}
> echo "foo bar"
> {noformat}
>  is split into 
> {noformat}
> ['echo', '"foo', 'bar"']
> {noformat}
> See: 
> https://github.com/apache/incubator-airflow/blob/master/airflow/configuration.py#L79



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-705][AIRFLOW-706] Fix run_command bugs

[xuanji]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 6bbf54154 -> 0bb6f2f45


[AIRFLOW-705][AIRFLOW-706] Fix run_command bugs

`run_command` currently uses `str.split` for shell
commands, but this
breaks commands like `echo "foo bar"`.  Instead,
use `shlex.split`,
which understands how to handle quotes.

In python3, subprocess functions return
stdout/stderr as `bytes`, not
`str`.  To fix py2/py3 compatibility,
`run_command` now decodes the
output so that it always returns a `str`.

Dear Airflow Maintainers,

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-705
-
https://issues.apache.org/jira/browse/AIRFLOW-706

Testing Done:
- Changed `sql_alchemy_conn` to
`sql_alchemy_conn_cmd` in test configuration to
exercise `run_command` in tests

Closes #2053 from thesquelched/AIRFLOW-706


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/0bb6f2f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/0bb6f2f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/0bb6f2f4

Branch: refs/heads/master
Commit: 0bb6f2f458ddf305d77fc63d5e817c9b6eb22270
Parents: 6bbf541
Author: Scott Kruger 
Authored: Thu Mar 16 19:43:01 2017 -0400
Committer: Li Xuanji 
Committed: Thu Mar 16 19:43:01 2017 -0400

--
 airflow/configuration.py |  7 +--
 tests/core.py| 16 +++-
 2 files changed, 20 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0bb6f2f4/airflow/configuration.py
--
diff --git a/airflow/configuration.py b/airflow/configuration.py
index fb3c11e..895c08d 100644
--- a/airflow/configuration.py
+++ b/airflow/configuration.py
@@ -24,6 +24,8 @@ import os
 import six
 import subprocess
 import warnings
+import shlex
+import sys
 
 from future import standard_library
 standard_library.install_aliases()
@@ -76,8 +78,9 @@ def run_command(command):
 Runs command and returns stdout
 """
 process = subprocess.Popen(
-command.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-output, stderr = process.communicate()
+shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+output, stderr = [stream.decode(sys.getdefaultencoding(), 'ignore')
+  for stream in process.communicate()]
 
 if process.returncode != 0:
 raise AirflowConfigException(

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0bb6f2f4/tests/core.py
--
diff --git a/tests/core.py b/tests/core.py
index e17920a..848553a 100644
--- a/tests/core.py
+++ b/tests/core.py
@@ -60,7 +60,7 @@ from airflow.utils.dates import infer_time_unit, round_time, 
scale_time_units
 from airflow.utils.logging import LoggingMixin
 from lxml import html
 from airflow.exceptions import AirflowException
-from airflow.configuration import AirflowConfigException
+from airflow.configuration import AirflowConfigException, run_command
 
 import six
 
@@ -1044,6 +1044,20 @@ class CoreTest(unittest.TestCase):
 session.commit()
 session.close()
 
+def test_run_command(self):
+if six.PY3:
+write = r'sys.stdout.buffer.write("\u1000foo".encode("utf8"))'
+else:
+write = r'sys.stdout.write(u"\u1000foo".encode("utf8"))'
+
+cmd = 'import sys; {0}; sys.stdout.flush()'.format(write)
+
+self.assertEqual(run_command("python -c '{0}'".format(cmd)),
+ u'\u1000foo' if six.PY3 else 'foo')
+
+self.assertEqual(run_command('echo "foo bar"'), u'foo bar\n')
+self.assertRaises(AirflowConfigException, run_command, 'bash -c "exit 
1"')
+
 
 class CliTests(unittest.TestCase):
 def setUp(self):

[jira] [Resolved] (AIRFLOW-963) Some code examples are not rendered in the airflow document

[Li Xuanji (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Xuanji resolved AIRFLOW-963.
---
   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2139
[https://github.com/apache/incubator-airflow/pull/2139]

> Some code examples are not rendered in the airflow document
> ---
>
> Key: AIRFLOW-963
> URL: https://issues.apache.org/jira/browse/AIRFLOW-963
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>  Labels: documentation
> Fix For: 1.9.0
>
> Attachments: Screen Shot 2017-03-10 at 16.06.24.png
>
>
> "Backfill and Catchup" section in ["Scheduling & Triggers" document| 
> http://airflow.apache.org/scheduler.html] says "In the example above, ...", 
> but that example isn't shown. See attached screenshot.
> Also, ["Security" document|http://airflow.apache.org/security.html] has the 
> same issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (AIRFLOW-990) DockerOperator fails when logging unicode string

[Jeremiah Lowin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Lowin resolved AIRFLOW-990.

   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2155
[https://github.com/apache/incubator-airflow/pull/2155]

> DockerOperator fails when logging unicode string
> 
>
> Key: AIRFLOW-990
> URL: https://issues.apache.org/jira/browse/AIRFLOW-990
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docker
>Affects Versions: Airflow 1.7.1
> Environment: Python 2.7
>Reporter: Vitor Baptista
>Assignee: Vitor Baptista
> Fix For: 1.9.0
>
>
> On line 
> https://github.com/apache/incubator-airflow/blob/1.7.1.3/airflow/operators/docker_operator.py#L164,
>  we're calling:
> {code:title=airflow/operators/docker_operator.py}
> for line in self.cli.logs(container=self.container['Id'], stream=True):
> logging.info("{}".format(line.strip()))
> {code}
> If `self.cli.logs()` return a string with a unicode character, this raises 
> the UnicodeDecodeError:
> {noformat}
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
> msg = self.format(record)
>   File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
> return fmt.format(record)
>   File "/usr/lib/python2.7/logging/__init__.py", line 476, in format
> raise e
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 0: 
> ordinal not in range(128)
> Logged from file docker_operator.py, line 165
> {noformat}
> A possible fix is to change that line to:
> {code:title=airflow/operators/docker_operator.py}
> for line in self.cli.logs(container=self.container['Id'], stream=True):
> logging.info(line.decode('utf-8').strip())
> {code}.
> This error doesn't happen on Python3. I haven't tested, but reading the code 
> it seems the same error exists on `master` as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-990) DockerOperator fails when logging unicode string

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929162#comment-15929162
 ] 

ASF subversion and git services commented on AIRFLOW-990:
-

Commit 6bbf54154316ce4a7375d56c1e8922135b863ae7 in incubator-airflow's branch 
refs/heads/master from [~vitorbaptista]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6bbf541 ]

[AIRFLOW-990] Fix Py27 unicode logging in DockerOperator

If a Docker container's logs has unicode values
(e.g. "echo "), the
DockerOperator will try to log it using
`logging.info(line)`, which raises a
`UnicodeDecodeError`. To solve this, we need to
decode the string as UTF-8
before sending it to `logging.info()`.

Closes #2155 from vitorbaptista/bug/fixes-
AIRFLOW-990


> DockerOperator fails when logging unicode string
> 
>
> Key: AIRFLOW-990
> URL: https://issues.apache.org/jira/browse/AIRFLOW-990
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docker
>Affects Versions: Airflow 1.7.1
> Environment: Python 2.7
>Reporter: Vitor Baptista
>Assignee: Vitor Baptista
> Fix For: 1.9.0
>
>
> On line 
> https://github.com/apache/incubator-airflow/blob/1.7.1.3/airflow/operators/docker_operator.py#L164,
>  we're calling:
> {code:title=airflow/operators/docker_operator.py}
> for line in self.cli.logs(container=self.container['Id'], stream=True):
> logging.info("{}".format(line.strip()))
> {code}
> If `self.cli.logs()` return a string with a unicode character, this raises 
> the UnicodeDecodeError:
> {noformat}
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
> msg = self.format(record)
>   File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
> return fmt.format(record)
>   File "/usr/lib/python2.7/logging/__init__.py", line 476, in format
> raise e
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 0: 
> ordinal not in range(128)
> Logged from file docker_operator.py, line 165
> {noformat}
> A possible fix is to change that line to:
> {code:title=airflow/operators/docker_operator.py}
> for line in self.cli.logs(container=self.container['Id'], stream=True):
> logging.info(line.decode('utf-8').strip())
> {code}.
> This error doesn't happen on Python3. I haven't tested, but reading the code 
> it seems the same error exists on `master` as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-963) Some code examples are not rendered in the airflow document

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929157#comment-15929157
 ] 

ASF subversion and git services commented on AIRFLOW-963:
-

Commit f69c1b4164dcb05c6e4361ca8e0f5567efbc2b10 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f69c1b4 ]

[AIRFLOW-963] Fix non-rendered code examples

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-963

Testing Done:
- ran sphinx-build locally and confirmed correctly
rendered

Closes #2139 from sekikn/AIRFLOW-963


> Some code examples are not rendered in the airflow document
> ---
>
> Key: AIRFLOW-963
> URL: https://issues.apache.org/jira/browse/AIRFLOW-963
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>  Labels: documentation
> Attachments: Screen Shot 2017-03-10 at 16.06.24.png
>
>
> "Backfill and Catchup" section in ["Scheduling & Triggers" document| 
> http://airflow.apache.org/scheduler.html] says "In the example above, ...", 
> but that example isn't shown. See attached screenshot.
> Also, ["Security" document|http://airflow.apache.org/security.html] has the 
> same issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-990] Fix Py27 unicode logging in DockerOperator

[jlowin]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master f69c1b416 -> 6bbf54154


[AIRFLOW-990] Fix Py27 unicode logging in DockerOperator

If a Docker container's logs has unicode values
(e.g. "echo 😁"), the
DockerOperator will try to log it using
`logging.info(line)`, which raises a
`UnicodeDecodeError`. To solve this, we need to
decode the string as UTF-8
before sending it to `logging.info()`.

Closes #2155 from vitorbaptista/bug/fixes-
AIRFLOW-990


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6bbf5415
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6bbf5415
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6bbf5415

Branch: refs/heads/master
Commit: 6bbf54154316ce4a7375d56c1e8922135b863ae7
Parents: f69c1b4
Author: Vitor Baptista 
Authored: Thu Mar 16 19:38:40 2017 -0400
Committer: Jeremiah Lowin 
Committed: Thu Mar 16 19:38:49 2017 -0400

--
 airflow/operators/docker_operator.py |  7 +--
 tests/operators/docker_operator.py   | 24 
 2 files changed, 29 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6bbf5415/airflow/operators/docker_operator.py
--
diff --git a/airflow/operators/docker_operator.py 
b/airflow/operators/docker_operator.py
index c75e776..2fd7484 100644
--- a/airflow/operators/docker_operator.py
+++ b/airflow/operators/docker_operator.py
@@ -175,14 +175,17 @@ class DockerOperator(BaseOperator):
 
 line = ''
 for line in self.cli.logs(container=self.container['Id'], 
stream=True):
-logging.info("{}".format(line.strip()))
+line = line.strip()
+if hasattr(line, 'decode'):
+line = line.decode('utf-8')
+logging.info(line)
 
 exit_code = self.cli.wait(self.container['Id'])
 if exit_code != 0:
 raise AirflowException('docker container failed')
 
 if self.xcom_push:
-return self.cli.logs(container=self.container['Id']) if 
self.xcom_all else str(line.strip())
+return self.cli.logs(container=self.container['Id']) if 
self.xcom_all else str(line)
 
 def get_command(self):
 if self.command is not None and self.command.strip().find('[') == 0:

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6bbf5415/tests/operators/docker_operator.py
--
diff --git a/tests/operators/docker_operator.py 
b/tests/operators/docker_operator.py
index cdfae94..2554aaa 100644
--- a/tests/operators/docker_operator.py
+++ b/tests/operators/docker_operator.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 import unittest
+import logging
 
 try:
 from airflow.operators.docker_operator import DockerOperator
@@ -103,6 +104,29 @@ class DockerOperatorTestCase(unittest.TestCase):
 
 @unittest.skipIf(mock is None, 'mock package not present')
 @mock.patch('airflow.operators.docker_operator.Client')
+def test_execute_unicode_logs(self, client_class_mock):
+client_mock = mock.Mock(spec=Client)
+client_mock.create_container.return_value = {'Id': 'some_id'}
+client_mock.create_host_config.return_value = mock.Mock()
+client_mock.images.return_value = []
+client_mock.logs.return_value = ['unicode container log 😁']
+client_mock.pull.return_value = []
+client_mock.wait.return_value = 0
+
+client_class_mock.return_value = client_mock
+
+originalRaiseExceptions = logging.raiseExceptions
+logging.raiseExceptions = True
+
+operator = DockerOperator(image='ubuntu', owner='unittest', 
task_id='unittest')
+
+with mock.patch('traceback.print_exception') as print_exception_mock:
+operator.execute(None)
+logging.raiseExceptions = originalRaiseExceptions
+print_exception_mock.assert_not_called()
+
+@unittest.skipIf(mock is None, 'mock package not present')
+@mock.patch('airflow.operators.docker_operator.Client')
 def test_execute_container_fails(self, client_class_mock):
 client_mock = mock.Mock(spec=Client)
 client_mock.create_container.return_value = {'Id': 'some_id'}

[jira] [Commented] (AIRFLOW-990) DockerOperator fails when logging unicode string

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929161#comment-15929161
 ] 

ASF subversion and git services commented on AIRFLOW-990:
-

Commit 6bbf54154316ce4a7375d56c1e8922135b863ae7 in incubator-airflow's branch 
refs/heads/master from [~vitorbaptista]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6bbf541 ]

[AIRFLOW-990] Fix Py27 unicode logging in DockerOperator

If a Docker container's logs has unicode values
(e.g. "echo "), the
DockerOperator will try to log it using
`logging.info(line)`, which raises a
`UnicodeDecodeError`. To solve this, we need to
decode the string as UTF-8
before sending it to `logging.info()`.

Closes #2155 from vitorbaptista/bug/fixes-
AIRFLOW-990


> DockerOperator fails when logging unicode string
> 
>
> Key: AIRFLOW-990
> URL: https://issues.apache.org/jira/browse/AIRFLOW-990
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docker
>Affects Versions: Airflow 1.7.1
> Environment: Python 2.7
>Reporter: Vitor Baptista
>Assignee: Vitor Baptista
> Fix For: 1.9.0
>
>
> On line 
> https://github.com/apache/incubator-airflow/blob/1.7.1.3/airflow/operators/docker_operator.py#L164,
>  we're calling:
> {code:title=airflow/operators/docker_operator.py}
> for line in self.cli.logs(container=self.container['Id'], stream=True):
> logging.info("{}".format(line.strip()))
> {code}
> If `self.cli.logs()` return a string with a unicode character, this raises 
> the UnicodeDecodeError:
> {noformat}
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
> msg = self.format(record)
>   File "/usr/lib/python2.7/logging/__init__.py", line 734, in format
> return fmt.format(record)
>   File "/usr/lib/python2.7/logging/__init__.py", line 476, in format
> raise e
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xf0 in position 0: 
> ordinal not in range(128)
> Logged from file docker_operator.py, line 165
> {noformat}
> A possible fix is to change that line to:
> {code:title=airflow/operators/docker_operator.py}
> for line in self.cli.logs(container=self.container['Id'], stream=True):
> logging.info(line.decode('utf-8').strip())
> {code}.
> This error doesn't happen on Python3. I haven't tested, but reading the code 
> it seems the same error exists on `master` as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-969) Catch bad python_callable argument at DAG construction rather than Task run

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929150#comment-15929150
 ] 

ASF subversion and git services commented on AIRFLOW-969:
-

Commit 12901ddfa9961a11feaa3f17696d19102ff8ecd0 in incubator-airflow's branch 
refs/heads/master from abloomston
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=12901dd ]

[AIRFLOW-969] Catch bad python_callable argument

Checks for callable when Operator is
created, not when it is run.

* added initial PythonOperator unit test, testing
run
* python_callable must be callable; added unit
test

Closes #2142 from abloomston/python-callable


> Catch bad python_callable argument at DAG construction rather than Task run
> ---
>
> Key: AIRFLOW-969
> URL: https://issues.apache.org/jira/browse/AIRFLOW-969
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: operators
>Reporter: Adam Bloomston
>Assignee: Adam Bloomston
>Priority: Minor
> Fix For: 1.9.0
>
>
> If a non-callable parameter for python_callable is passed to PythonOperator, 
> it should fail to instantiate.  This will move such failures from task run to 
> DAG instantiation. Better to catch such errors sooner rather than later in 
> execution.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-969] Catch bad python_callable argument

[jlowin]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master a8bd1695e -> 12901ddfa


[AIRFLOW-969] Catch bad python_callable argument

Checks for callable when Operator is
created, not when it is run.

* added initial PythonOperator unit test, testing
run
* python_callable must be callable; added unit
test

Closes #2142 from abloomston/python-callable


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/12901ddf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/12901ddf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/12901ddf

Branch: refs/heads/master
Commit: 12901ddfa9961a11feaa3f17696d19102ff8ecd0
Parents: a8bd169
Author: abloomston 
Authored: Thu Mar 16 19:36:00 2017 -0400
Committer: Jeremiah Lowin 
Committed: Thu Mar 16 19:36:08 2017 -0400

--
 airflow/operators/python_operator.py |  3 ++
 tests/operators/__init__.py  |  1 +
 tests/operators/python_operator.py   | 79 +++
 3 files changed, 83 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/12901ddf/airflow/operators/python_operator.py
--
diff --git a/airflow/operators/python_operator.py 
b/airflow/operators/python_operator.py
index b5f6386..a17e6fa 100644
--- a/airflow/operators/python_operator.py
+++ b/airflow/operators/python_operator.py
@@ -16,6 +16,7 @@ from builtins import str
 from datetime import datetime
 import logging
 
+from airflow.exceptions import AirflowException
 from airflow.models import BaseOperator, TaskInstance
 from airflow.utils.state import State
 from airflow.utils.decorators import apply_defaults
@@ -63,6 +64,8 @@ class PythonOperator(BaseOperator):
 templates_exts=None,
 *args, **kwargs):
 super(PythonOperator, self).__init__(*args, **kwargs)
+if not callable(python_callable):
+raise AirflowException('`python_callable` param must be callable')
 self.python_callable = python_callable
 self.op_args = op_args or []
 self.op_kwargs = op_kwargs or {}

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/12901ddf/tests/operators/__init__.py
--
diff --git a/tests/operators/__init__.py b/tests/operators/__init__.py
index 1fb0e5e..7a517a1 100644
--- a/tests/operators/__init__.py
+++ b/tests/operators/__init__.py
@@ -18,3 +18,4 @@ from .operators import *
 from .sensors import *
 from .hive_operator import *
 from .s3_to_hive_operator import *
+from .python_operator import *

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/12901ddf/tests/operators/python_operator.py
--
diff --git a/tests/operators/python_operator.py 
b/tests/operators/python_operator.py
new file mode 100644
index 000..621172f
--- /dev/null
+++ b/tests/operators/python_operator.py
@@ -0,0 +1,79 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function, unicode_literals
+
+import datetime
+import unittest
+
+from airflow import configuration, DAG
+from airflow.operators.python_operator import PythonOperator
+
+from airflow.exceptions import AirflowException
+
+DEFAULT_DATE = datetime.datetime(2016, 1, 1)
+END_DATE = datetime.datetime(2016, 1, 2)
+INTERVAL = datetime.timedelta(hours=12)
+FROZEN_NOW = datetime.datetime(2016, 1, 2, 12, 1, 1)
+
+
+class PythonOperatorTest(unittest.TestCase):
+
+def setUp(self):
+super(PythonOperatorTest, self).setUp()
+configuration.load_test_config()
+self.dag = DAG(
+'test_dag',
+default_args={
+'owner': 'airflow',
+'start_date': DEFAULT_DATE},
+schedule_interval=INTERVAL)
+self.addCleanup(self.dag.clear)
+self.clear_run()
+self.addCleanup(self.clear_run)
+
+def do_run(self):
+self.run = True
+
+def clear_run(self):
+self.run = False
+
+def is_run(self):
+return self.run
+
+def test_python_operator_run(self):
+"""Tests that the 

[jira] [Commented] (AIRFLOW-963) Some code examples are not rendered in the airflow document

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929156#comment-15929156
 ] 

ASF subversion and git services commented on AIRFLOW-963:
-

Commit f69c1b4164dcb05c6e4361ca8e0f5567efbc2b10 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f69c1b4 ]

[AIRFLOW-963] Fix non-rendered code examples

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-963

Testing Done:
- ran sphinx-build locally and confirmed correctly
rendered

Closes #2139 from sekikn/AIRFLOW-963


> Some code examples are not rendered in the airflow document
> ---
>
> Key: AIRFLOW-963
> URL: https://issues.apache.org/jira/browse/AIRFLOW-963
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>  Labels: documentation
> Attachments: Screen Shot 2017-03-10 at 16.06.24.png
>
>
> "Backfill and Catchup" section in ["Scheduling & Triggers" document| 
> http://airflow.apache.org/scheduler.html] says "In the example above, ...", 
> but that example isn't shown. See attached screenshot.
> Also, ["Security" document|http://airflow.apache.org/security.html] has the 
> same issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-963] Fix non-rendered code examples

[xuanji]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 12901ddfa -> f69c1b416


[AIRFLOW-963] Fix non-rendered code examples

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-963

Testing Done:
- ran sphinx-build locally and confirmed correctly
rendered

Closes #2139 from sekikn/AIRFLOW-963


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f69c1b41
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f69c1b41
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f69c1b41

Branch: refs/heads/master
Commit: f69c1b4164dcb05c6e4361ca8e0f5567efbc2b10
Parents: 12901dd
Author: Kengo Seki 
Authored: Thu Mar 16 19:37:23 2017 -0400
Committer: Li Xuanji 
Committed: Thu Mar 16 19:37:23 2017 -0400

--
 docs/scheduler.rst | 1 +
 docs/security.rst  | 1 +
 2 files changed, 2 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f69c1b41/docs/scheduler.rst
--
diff --git a/docs/scheduler.rst b/docs/scheduler.rst
index 749d58a..efc98a8 100644
--- a/docs/scheduler.rst
+++ b/docs/scheduler.rst
@@ -91,6 +91,7 @@ will do, is to instruct the scheduler to only create a DAG 
Run for the most curr
 interval series.
 
 .. code:: python
+
 """
 Code that goes along with the Airflow tutorial located at:
 
https://github.com/airbnb/airflow/blob/master/airflow/example_dags/tutorial.py

http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f69c1b41/docs/security.rst
--
diff --git a/docs/security.rst b/docs/security.rst
index 70db606..7c06fe3 100644
--- a/docs/security.rst
+++ b/docs/security.rst
@@ -324,6 +324,7 @@ this, assuming as airflow is running as the `airflow` user. 
Note that this means
 the airflow user must be trusted and treated the same way as the root user.
 
 .. code-block:: none
+
 airflow ALL=(ALL) NOPASSWD: ALL
 
 Subtasks with impersonation will still log to the same folder, except that the 
files they

[jira] [Commented] (AIRFLOW-963) Some code examples are not rendered in the airflow document

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929155#comment-15929155
 ] 

ASF subversion and git services commented on AIRFLOW-963:
-

Commit f69c1b4164dcb05c6e4361ca8e0f5567efbc2b10 in incubator-airflow's branch 
refs/heads/master from [~sekikn]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f69c1b4 ]

[AIRFLOW-963] Fix non-rendered code examples

Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-963

Testing Done:
- ran sphinx-build locally and confirmed correctly
rendered

Closes #2139 from sekikn/AIRFLOW-963


> Some code examples are not rendered in the airflow document
> ---
>
> Key: AIRFLOW-963
> URL: https://issues.apache.org/jira/browse/AIRFLOW-963
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: docs
>Reporter: Kengo Seki
>Assignee: Kengo Seki
>Priority: Minor
>  Labels: documentation
> Attachments: Screen Shot 2017-03-10 at 16.06.24.png
>
>
> "Backfill and Catchup" section in ["Scheduling & Triggers" document| 
> http://airflow.apache.org/scheduler.html] says "In the example above, ...", 
> but that example isn't shown. See attached screenshot.
> Also, ["Security" document|http://airflow.apache.org/security.html] has the 
> same issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-984) Subdags unrecognized when subclassing SubDagOperator

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929145#comment-15929145
 ] 

ASF subversion and git services commented on AIRFLOW-984:
-

Commit a8bd1695e44e56375a399a9c7a0af0b137922e24 in incubator-airflow's branch 
refs/heads/master from [~patrickmckenna]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=a8bd169 ]

[AIRFLOW-984] Enable subclassing of SubDagOperator

Note this maintains an awkward name check
for backwards compatibility reasons.

Closes #2152 from patrickmckenna/recognize-subdag-
subclasses


> Subdags unrecognized when subclassing SubDagOperator
> 
>
> Key: AIRFLOW-984
> URL: https://issues.apache.org/jira/browse/AIRFLOW-984
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Patrick McKenna
>Priority: Minor
> Fix For: 1.9.0
>
>
> If a user subclasses SubDagOperator, the parent DAG will not pick up the 
> subdags:
> https://github.com/apache/incubator-airflow/blob/c44e2009ee625ce4a82c50e585a3c8617d9b4ff8/airflow/models.py#L2974
> which means a DagBag won't find them:
> https://github.com/apache/incubator-airflow/blob/c44e2009ee625ce4a82c50e585a3c8617d9b4ff8/airflow/models.py#L311
> https://github.com/apache/incubator-airflow/blob/c44e2009ee625ce4a82c50e585a3c8617d9b4ff8/airflow/models.py#L365
> This PR appears to be the cause: 
> https://github.com/apache/incubator-airflow/pull/1196/files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-984] Enable subclassing of SubDagOperator

[jlowin]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master e08b102c0 -> a8bd1695e


[AIRFLOW-984] Enable subclassing of SubDagOperator

Note this maintains an awkward name check
for backwards compatibility reasons.

Closes #2152 from patrickmckenna/recognize-subdag-
subclasses


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a8bd1695
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a8bd1695
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a8bd1695

Branch: refs/heads/master
Commit: a8bd1695e44e56375a399a9c7a0af0b137922e24
Parents: e08b102
Author: Patrick McKenna 
Authored: Thu Mar 16 19:33:31 2017 -0400
Committer: Jeremiah Lowin 
Committed: Thu Mar 16 19:33:38 2017 -0400

--
 airflow/models.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a8bd1695/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index ad3346a..561b002 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2972,11 +2972,12 @@ class DAG(BaseDag, LoggingMixin):
 """
 # Check SubDag for class but don't check class directly, see
 # https://github.com/airbnb/airflow/issues/1168
+from airflow.operators.subdag_operator import SubDagOperator
 l = []
 for task in self.tasks:
-if (
-task.__class__.__name__ == 'SubDagOperator' and
-hasattr(task, 'subdag')):
+if (isinstance(task, SubDagOperator) or
+#TODO remove in Airflow 2.0
+type(task).__name__ == 'SubDagOperator'):
 l.append(task.subdag)
 l += task.subdag.subdags
 return l

[jira] [Resolved] (AIRFLOW-984) Subdags unrecognized when subclassing SubDagOperator

[Jeremiah Lowin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Lowin resolved AIRFLOW-984.

   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2152
[https://github.com/apache/incubator-airflow/pull/2152]

> Subdags unrecognized when subclassing SubDagOperator
> 
>
> Key: AIRFLOW-984
> URL: https://issues.apache.org/jira/browse/AIRFLOW-984
> Project: Apache Airflow
>  Issue Type: Bug
>Reporter: Patrick McKenna
>Priority: Minor
> Fix For: 1.9.0
>
>
> If a user subclasses SubDagOperator, the parent DAG will not pick up the 
> subdags:
> https://github.com/apache/incubator-airflow/blob/c44e2009ee625ce4a82c50e585a3c8617d9b4ff8/airflow/models.py#L2974
> which means a DagBag won't find them:
> https://github.com/apache/incubator-airflow/blob/c44e2009ee625ce4a82c50e585a3c8617d9b4ff8/airflow/models.py#L311
> https://github.com/apache/incubator-airflow/blob/c44e2009ee625ce4a82c50e585a3c8617d9b4ff8/airflow/models.py#L365
> This PR appears to be the cause: 
> https://github.com/apache/incubator-airflow/pull/1196/files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (AIRFLOW-997) Change setup.cfg to point to Apache instead of Max

[Arthur Wiedmer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arthur Wiedmer resolved AIRFLOW-997.

   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2162
[https://github.com/apache/incubator-airflow/pull/2162]

> Change setup.cfg to point to Apache instead of Max
> --
>
> Key: AIRFLOW-997
> URL: https://issues.apache.org/jira/browse/AIRFLOW-997
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
> Fix For: 1.9.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[1/2] incubator-airflow git commit: [AIRFLOW-997] Update setup.cfg to point to Apache

[arthur]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master ca1623386 -> e08b102c0


[AIRFLOW-997] Update setup.cfg to point to Apache

The setup.cfg should point to the Apache PMC as "author" and the
dev mailing list as contact email.


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/75cd460c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/75cd460c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/75cd460c

Branch: refs/heads/master
Commit: 75cd460ce8e557c44c0c79245105fd535b037210
Parents: 3d6095f
Author: Arthur Wiedmer 
Authored: Thu Mar 16 13:35:05 2017 -0700
Committer: Arthur Wiedmer 
Committed: Thu Mar 16 13:35:13 2017 -0700

--
 setup.cfg | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/75cd460c/setup.cfg
--
diff --git a/setup.cfg b/setup.cfg
index 0dd2c39..76ddf37 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -15,8 +15,8 @@
 name = Airflow
 summary = Airflow is a system to programmatically author, schedule and monitor 
data pipelines.
 description-file = README.md
-author = Maxime Beauchemin
-author-email = maximebeauche...@gmail.com
+author = Apache Airflow PMC
+author-email = d...@airflow.incubator.apache.org
 license = Apache License, Version 2.0
 
 [files]

[jira] [Commented] (AIRFLOW-997) Change setup.cfg to point to Apache instead of Max

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928944#comment-15928944
 ] 

ASF subversion and git services commented on AIRFLOW-997:
-

Commit 75cd460ce8e557c44c0c79245105fd535b037210 in incubator-airflow's branch 
refs/heads/master from [~artwr]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=75cd460 ]

[AIRFLOW-997] Update setup.cfg to point to Apache

The setup.cfg should point to the Apache PMC as "author" and the
dev mailing list as contact email.


> Change setup.cfg to point to Apache instead of Max
> --
>
> Key: AIRFLOW-997
> URL: https://issues.apache.org/jira/browse/AIRFLOW-997
> Project: Apache Airflow
>  Issue Type: Improvement
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[2/2] incubator-airflow git commit: Merge pull request #2162 from artwr/artwr-update_setup_cfg

[arthur]

Merge pull request #2162 from artwr/artwr-update_setup_cfg


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e08b102c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e08b102c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e08b102c

Branch: refs/heads/master
Commit: e08b102c002eb0f8f08229063738f9a94a186278
Parents: ca16233 75cd460
Author: Arthur Wiedmer 
Authored: Thu Mar 16 14:38:23 2017 -0700
Committer: Arthur Wiedmer 
Committed: Thu Mar 16 14:38:23 2017 -0700

--
 setup.cfg | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
--

[jira] [Created] (AIRFLOW-997) Change setup.cfg to point to Apache instead of Max

[Arthur Wiedmer (JIRA)]

Arthur Wiedmer created AIRFLOW-997:
--

 Summary: Change setup.cfg to point to Apache instead of Max
 Key: AIRFLOW-997
 URL: https://issues.apache.org/jira/browse/AIRFLOW-997
 Project: Apache Airflow
  Issue Type: Improvement
Reporter: Arthur Wiedmer
Assignee: Arthur Wiedmer
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (AIRFLOW-994) Add MiNODES to the AIRFLOW Active Users List

[Jeremiah Lowin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Lowin resolved AIRFLOW-994.

   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2159
[https://github.com/apache/incubator-airflow/pull/2159]

> Add MiNODES to the AIRFLOW Active Users List
> 
>
> Key: AIRFLOW-994
> URL: https://issues.apache.org/jira/browse/AIRFLOW-994
> Project: Apache Airflow
>  Issue Type: Task
>Reporter: Alexander Mueller
>Assignee: Alexander Mueller
>Priority: Trivial
> Fix For: 1.9.0
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> Please add MiNODES to the Active Users list: Please use their Website 
> https://www.minodes.com and their airflow users '@dice89' and 'diazcelsa'.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-994] Add MiNODES to the official airflow user list

[jlowin]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 0473ba206 -> ca1623386


[AIRFLOW-994] Add MiNODES to the official airflow user list

Add MiNODES to the official airflow user list

Closes #2159 from dice89/master


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ca162338
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ca162338
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ca162338

Branch: refs/heads/master
Commit: ca162338697ae6f7d6287f4d2c1f6544b77e077f
Parents: 0473ba2
Author: Alexander Mueller 
Authored: Thu Mar 16 16:18:28 2017 -0400
Committer: Jeremiah Lowin 
Committed: Thu Mar 16 16:18:32 2017 -0400

--
 README.md | 1 +
 1 file changed, 1 insertion(+)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ca162338/README.md
--
diff --git a/README.md b/README.md
index fb268c9..982e5cc 100644
--- a/README.md
+++ b/README.md
@@ -129,6 +129,7 @@ Currently **officially** using Airflow:
 1. [Madrone](http://madroneco.com/) 
[[@mbreining](https://github.com/mbreining) & 
[@scotthb](https://github.com/scotthb)]
 1. [Markovian](https://markovian.com/) [[@al-xv](https://github.com/al-xv), 
[@skogsbaeck](https://github.com/skogsbaeck), 
[@waltherg](https://github.com/waltherg)]
 1. [Mercadoni](https://www.mercadoni.com.co) 
[[@demorenoc](https://github.com/demorenoc)]
+1. [MiNODES](https://www.minodes.com) [[@dice89](https://github.com/dice89), 
[@diazcelsa](https://github.com/diazcelsa)]
 1. [MFG Labs](https://github.com/MfgLabs)
 1. [mytaxi](https://mytaxi.com) [[@mytaxi](https://github.com/mytaxi)]
 1. [Nerdwallet](https://www.nerdwallet.com)

[jira] [Resolved] (AIRFLOW-995) Update Github PR template

[Jeremiah Lowin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/AIRFLOW-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremiah Lowin resolved AIRFLOW-995.

   Resolution: Fixed
Fix Version/s: 1.9.0

Issue resolved by pull request #2160
[https://github.com/apache/incubator-airflow/pull/2160]

> Update Github PR template
> -
>
> Key: AIRFLOW-995
> URL: https://issues.apache.org/jira/browse/AIRFLOW-995
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>Priority: Minor
> Fix For: 1.9.0
>
>
> The Github PR template looks great when rendered but is harder to parse while 
> in editing mode (which is how all PR authors initially see it). A new 
> template would be clear whether editing or previewing and include checkboxes 
> for force some user acknowledgement of each required step.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-1) Migrate GitHub code to Apache git

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928790#comment-15928790
 ] 

ASF subversion and git services commented on AIRFLOW-1:
---

Commit 0473ba2068fc385ec79eb01f323500465d934ec8 in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0473ba2 ]

[AIRFLOW-995][AIRFLOW-1] Update GitHub PR Template

Closes #2160 from jlowin/PR-template


> Migrate GitHub code to Apache git
> -
>
> Key: AIRFLOW-1
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: project-management
>Reporter: Maxime Beauchemin
>Assignee: Maxime Beauchemin
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-995) Update Github PR template

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928789#comment-15928789
 ] 

ASF subversion and git services commented on AIRFLOW-995:
-

Commit 0473ba2068fc385ec79eb01f323500465d934ec8 in incubator-airflow's branch 
refs/heads/master from [~jlowin]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=0473ba2 ]

[AIRFLOW-995][AIRFLOW-1] Update GitHub PR Template

Closes #2160 from jlowin/PR-template


> Update Github PR template
> -
>
> Key: AIRFLOW-995
> URL: https://issues.apache.org/jira/browse/AIRFLOW-995
> Project: Apache Airflow
>  Issue Type: Improvement
>Affects Versions: 1.8.0
>Reporter: Jeremiah Lowin
>Assignee: Jeremiah Lowin
>Priority: Minor
>
> The Github PR template looks great when rendered but is harder to parse while 
> in editing mode (which is how all PR authors initially see it). A new 
> template would be clear whether editing or previewing and include checkboxes 
> for force some user acknowledgement of each required step.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

incubator-airflow git commit: [AIRFLOW-995][AIRFLOW-1] Update GitHub PR Template

[jlowin]

Repository: incubator-airflow
Updated Branches:
  refs/heads/master 3d6095ff5 -> 0473ba206


[AIRFLOW-995][AIRFLOW-1] Update GitHub PR Template

Closes #2160 from jlowin/PR-template


Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/0473ba20
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/0473ba20
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/0473ba20

Branch: refs/heads/master
Commit: 0473ba2068fc385ec79eb01f323500465d934ec8
Parents: 3d6095f
Author: Jeremiah Lowin 
Authored: Thu Mar 16 16:14:15 2017 -0400
Committer: Jeremiah Lowin 
Committed: Thu Mar 16 16:14:15 2017 -0400

--
 .github/PULL_REQUEST_TEMPLATE.md | 58 ---
 1 file changed, 34 insertions(+), 24 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0473ba20/.github/PULL_REQUEST_TEMPLATE.md
--
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index b92e29a..83bd1fc 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,25 +1,35 @@
-Please accept this PR that addresses the following issues:
-- *(MANDATORY - replace with a link to JIRA - e.g. 
https://issues.apache.org/jira/browse/AIRFLOW-XXX)*
-
-Testing Done:
-- Unittests are required, if you do not include new unit tests please
-specify why you think this is not required. We like to improve our
-coverage so a non existing test is even a better reason to include one.
-
-Reminders for contributors (REQUIRED!):
-* Your PR's title must reference an issue on 
-[Airflow's JIRA](https://issues.apache.org/jira/browse/AIRFLOW/). 
-For example, a PR called "[AIRFLOW-1] My Amazing PR" would close JIRA 
-issue #1. Please open a new issue if required!
-
-* For all PRs with UI changes, you must provide screenshots. If the UI changes 
are not obvious, either annotate the images or provide before/after screenshots.
-
-* Please squash your commits when possible and follow the [How to write a good 
git commit message](http://chris.beams.io/posts/git-commit/). 
-Summarized as follows:
-  1. Separate subject from body with a blank line
-  2. Limit the subject line to 50 characters
-  3. Do not end the subject line with a period
-  4. Use the imperative mood in the subject line (add, not adding)
-  5. Wrap the body at 72 characters
-  6. Use the body to explain what and why vs. how
+Dear Airflow maintainers,
+
+Please accept this PR. I understand that it will not be reviewed until I have 
checked off all the steps below!
+
+### Opened PR
+- [x] Opened a PR on Github
+
+
+### [JIRA](https://issues.apache.org/jira/browse/AIRFLOW/)
+- [ ] My PR addresses the following Airflow JIRA issues:
+- https://issues.apache.org/jira/browse/AIRFLOW-XXX
+- [ ] The PR title references the JIRA issues. For example, "[AIRFLOW-1] My 
Airflow PR"
+
+
+### Tests
+- [ ] My PR adds unit tests
+- [ ] __OR__ my PR does not need testing for this extremely good reason:
+
+
+### Description
+- [ ] Here are some details about my PR:
+- [ ] Here are screenshots of any UI changes, if appropriate:
+
+
+### Commits
+- [ ] Each commit subject references a JIRA issue. For example, "[AIRFLOW-1] 
Add new feature"
+- [ ] Multiple commits addressing the same JIRA issue have been squashed
+- [ ] My commits follow the guidelines from "[How to write a good git commit 
message](http://chris.beams.io/posts/git-commit/)":
+  1. Subject is separated from body by a blank line
+  2. Subject is limited to 50 characters
+  3. Subject does not end with a period
+  4. Subject uses the imperative mood ("add", not "adding")
+  5. Body wraps at 72 characters
+  6. Body explains "what" and "why", not "how"
 

[jira] [Commented] (AIRFLOW-996) Version.py is out of date for 1.8-stable

[Christian Petro (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928594#comment-15928594
 ] 

Christian Petro commented on AIRFLOW-996:
-

Created PR #2161(https://github.com/apache/incubator-airflow/pull/2161)

> Version.py is out of date for 1.8-stable
> 
>
> Key: AIRFLOW-996
> URL: https://issues.apache.org/jira/browse/AIRFLOW-996
> Project: Apache Airflow
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Christian Petro
>Assignee: Christian Petro
>Priority: Trivial
>
> Version.py is out of date, update to match tag



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-996) Version.py is out of date for 1.8-stable

[Christian Petro (JIRA)]

Christian Petro created AIRFLOW-996:
---

 Summary: Version.py is out of date for 1.8-stable
 Key: AIRFLOW-996
 URL: https://issues.apache.org/jira/browse/AIRFLOW-996
 Project: Apache Airflow
  Issue Type: Bug
Affects Versions: 1.8.0
Reporter: Christian Petro
Assignee: Christian Petro
Priority: Trivial


Version.py is out of date, update to match tag



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (AIRFLOW-115) Migrate and Refactor AWS integration to use boto3 and better structured hooks

[Adamos Loizou (JIRA)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928529#comment-15928529
 ] 

Adamos Loizou commented on AIRFLOW-115:
---

+1 on this one.

Airflow's S3Hook is also problematic when using non-US S3 buckets such as 
eu-central-1 which also require v4 S3 communication API.

Because of this we had to scrap using Airflow plugins and code it directly 
ourselves as a python app.

It would be great if Airflow could upgrade boto and allow for more 
configuration to be passed down to the plugin.

> Migrate and Refactor AWS integration to use boto3 and better structured hooks
> -
>
> Key: AIRFLOW-115
> URL: https://issues.apache.org/jira/browse/AIRFLOW-115
> Project: Apache Airflow
>  Issue Type: Improvement
>  Components: aws, boto3, hooks
>Reporter: Arthur Wiedmer
>Assignee: Arthur Wiedmer
>Priority: Minor
>
> h2. Current State
> The current AWS integration is mostly done through the S3Hook, which uses non 
> standard credentials parsing on top of using boto instead of boto3 which is 
> the current supported AWS sdk for Python.
> h2. Proposal
> an AWSHook should be provided that maps Airflow connections to the boto3 API. 
> Operators working with s3, as well as other AWS services would then inherit 
> from this hook but extend the functionality with service specific methods 
> like get_key for S3, start_cluster for EMR, enqueue for SQS, send_email for 
> SES etc...
> * AWSHook
> ** S3Hook
> ** EMRHook
> ** SQSHook
> ** SESHook
> ...
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[incubator-airflow] Git Push Summary

[bolke]

Repository: incubator-airflow
Updated Tags:  refs/tags/1.8.0 [created] 76602b205

[jira] [Created] (AIRFLOW-995) Update Github PR template

[Jeremiah Lowin (JIRA)]

Jeremiah Lowin created AIRFLOW-995:
--

 Summary: Update Github PR template
 Key: AIRFLOW-995
 URL: https://issues.apache.org/jira/browse/AIRFLOW-995
 Project: Apache Airflow
  Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: Jeremiah Lowin
Assignee: Jeremiah Lowin
Priority: Minor


The Github PR template looks great when rendered but is harder to parse while 
in editing mode (which is how all PR authors initially see it). A new template 
would be clear whether editing or previewing and include checkboxes for force 
some user acknowledgement of each required step.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (AIRFLOW-994) Add MiNODES to the AIRFLOW Active Users List

[Alexander Mueller (JIRA)]

Alexander Mueller created AIRFLOW-994:
-

 Summary: Add MiNODES to the AIRFLOW Active Users List
 Key: AIRFLOW-994
 URL: https://issues.apache.org/jira/browse/AIRFLOW-994
 Project: Apache Airflow
  Issue Type: Task
Reporter: Alexander Mueller
Assignee: Alexander Mueller
Priority: Trivial


Please add MiNODES to the Active Users list: Please use their Website 
https://www.minodes.com and their airflow users '@dice89' and 'diazcelsa'.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)