[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.7.1.3-docs [created] 209bf9c7b
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.6.2-docs [created] 35cb43992
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.7.1.3 c643a553c -> 209bf9c7b
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.7.0rc1 [created] ffc070100 refs/tags/1.7.1rc1 [created] 658185870 refs/tags/airbnb_1.7.1rc1 [created] 658185870 refs/tags/airbnb_1.7.1rc10 [created] 8d7297573 refs/tags/airbnb_1.7.1rc11 [created] 39ab764a9 refs/tags/airbnb_1.7.1rc13 [created] 15637b0ed refs/tags/airbnb_1.7.1rc14 [created] 061dfc6cc refs/tags/airbnb_1.7.1rc2 [created] 173b19313 refs/tags/airbnb_1.7.1rc3 [created] f1ff65c8c refs/tags/airbnb_1.7.1rc4 [created] 173b19313 refs/tags/airbnb_1.7.1rc6 [created] 563be1324 refs/tags/airbnb_1.7.1rc7 [created] f4e9fd68e refs/tags/airbnb_1.7.1rc8 [created] 51ee7d36c refs/tags/airbnb_prod.1.6.1.0 [created] 546e54cef refs/tags/airbnb_prod.1.6.1.1 [created] 190ee9398 refs/tags/airbnb_prod.1.6.1.2 [created] 3be5e8115 refs/tags/airbnb_prod.1.6.1.3 [created] 0f69d352e refs/tags/airbnb_prod.1.6.1.4 [created] a0e41d32e refs/tags/airbnb_prod.1.6.1.5 [created] 715b98509 refs/tags/airbnb_prod.1.6.1.6 [created] e19bc21ff refs/tags/airbnb_prod.1.6.1.7 [created] 101c95782 refs/tags/airbnb_prod.1.6.1.8 [created] 072f4e8d2 refs/tags/airbnb_prod.1.6.2.1 [created] 85ff87ee1 refs/tags/airbnb_prod.1.6.2.10 [created] ea16895a0 refs/tags/airbnb_prod.1.6.2.2 [created] d233e41da refs/tags/airbnb_prod.1.6.2.3 [created] bd0966069 refs/tags/airbnb_prod.1.6.2.4 [created] 2c150966f refs/tags/airbnb_prod.1.6.2.5 [created] 0f5ff784d refs/tags/airbnb_prod.1.6.2.6 [created] 95fa58fbd refs/tags/airbnb_prod.1.6.2.7 [created] f74e6900e refs/tags/airbnb_prod.1.6.2.8 [created] 7866d5d32 refs/tags/airbnb_prod.1.6.2.9 [created] ffc070100
[jira] [Resolved] (AIRFLOW-954) Installing future 0.16 breaks airflow initdb on Python 2.7
[
https://issues.apache.org/jira/browse/AIRFLOW-954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maxime Beauchemin resolved AIRFLOW-954.
---
Resolution: Fixed
> Installing future 0.16 breaks airflow initdb on Python 2.7
> --
>
> Key: AIRFLOW-954
> URL: https://issues.apache.org/jira/browse/AIRFLOW-954
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Python 2.7.3 on Ubuntu 12.04 with a clean virtualenv and
> HEAD of airflow master installed
>Reporter: Sean Cronin
>
> On master, if you run {{airflow initdb}} after installing HEAD of airflow
> master with Python 2.7, you get the following error:
> {code}
> Traceback (most recent call last):
> File "/home/sean/.virtualenvs/airflow-test/bin/airflow", line 17, in
>
> from airflow import configuration
> File
> "/home/sean/.virtualenvs/airflow-test/local/lib/python2.7/site-packages/airflow/__init__.py",
> line 29, in
> from airflow import configuration as conf
> File
> "/home/sean/.virtualenvs/airflow-test/local/lib/python2.7/site-packages/airflow/configuration.py",
> line 33, in
> from configparser import ConfigParser
> ImportError: No module named configparser
> {code}
> This seems to be due to https://github.com/apache/incubator-airflow/pull/2091
> which bumps {{future}} in {{setup.py}} so that future 0.16 can be installed.
> Python future [got rid
> of](http://python-future.org/whatsnew.html#what-s-new-in-version-0-16-0-2016-10-27)
> their {{configparser}} alias in 0.16
> The recommended way to fix this is to install {{future==0.16}} and install
> {{configparser}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (AIRFLOW-954) Installing future 0.16 breaks airflow initdb on Python 2.7
[
https://issues.apache.org/jira/browse/AIRFLOW-954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902503#comment-15902503
]
ASF subversion and git services commented on AIRFLOW-954:
-
Commit f3924696ff32942abc8c5709876c5717e0a82445 in incubator-airflow's branch
refs/heads/master from [~seancron]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f392469 ]
[AIRFLOW-954] Fix configparser ImportError
Fixes support for Python 2.7 since
https://github.com/apache/incubator-airflow/pull/2091 was merged
> Installing future 0.16 breaks airflow initdb on Python 2.7
> --
>
> Key: AIRFLOW-954
> URL: https://issues.apache.org/jira/browse/AIRFLOW-954
> Project: Apache Airflow
> Issue Type: Bug
> Environment: Python 2.7.3 on Ubuntu 12.04 with a clean virtualenv and
> HEAD of airflow master installed
>Reporter: Sean Cronin
>
> On master, if you run {{airflow initdb}} after installing HEAD of airflow
> master with Python 2.7, you get the following error:
> {code}
> Traceback (most recent call last):
> File "/home/sean/.virtualenvs/airflow-test/bin/airflow", line 17, in
>
> from airflow import configuration
> File
> "/home/sean/.virtualenvs/airflow-test/local/lib/python2.7/site-packages/airflow/__init__.py",
> line 29, in
> from airflow import configuration as conf
> File
> "/home/sean/.virtualenvs/airflow-test/local/lib/python2.7/site-packages/airflow/configuration.py",
> line 33, in
> from configparser import ConfigParser
> ImportError: No module named configparser
> {code}
> This seems to be due to https://github.com/apache/incubator-airflow/pull/2091
> which bumps {{future}} in {{setup.py}} so that future 0.16 can be installed.
> Python future [got rid
> of](http://python-future.org/whatsnew.html#what-s-new-in-version-0-16-0-2016-10-27)
> their {{configparser}} alias in 0.16
> The recommended way to fix this is to install {{future==0.16}} and install
> {{configparser}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[2/2] incubator-airflow git commit: Merge pull request #2130 from seancron/airflow-954
Merge pull request #2130 from seancron/airflow-954 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/abbb4ee5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/abbb4ee5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/abbb4ee5 Branch: refs/heads/master Commit: abbb4ee5cd02c691c611631a79212806f77c0fb2 Parents: d7f8334 f392469 Author: Maxime Beauchemin Authored: Wed Mar 8 21:08:00 2017 -0800 Committer: Maxime Beauchemin Committed: Wed Mar 8 21:08:00 2017 -0800 -- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --
[1/2] incubator-airflow git commit: [AIRFLOW-954] Fix configparser ImportError
Repository: incubator-airflow Updated Branches: refs/heads/master d7f8334d2 -> abbb4ee5c [AIRFLOW-954] Fix configparser ImportError Fixes support for Python 2.7 since https://github.com/apache/incubator-airflow/pull/2091 was merged Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f3924696 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f3924696 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f3924696 Branch: refs/heads/master Commit: f3924696ff32942abc8c5709876c5717e0a82445 Parents: e79dee8 Author: Sean Cronin Authored: Tue Mar 7 20:52:48 2017 -0500 Committer: Sean Cronin Committed: Tue Mar 7 20:52:48 2017 -0500 -- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3924696/setup.py -- diff --git a/setup.py b/setup.py index 87808d6..481d427 100644 --- a/setup.py +++ b/setup.py @@ -202,6 +202,7 @@ def do_setup(): scripts=['airflow/bin/airflow'], install_requires=[ 'alembic>=0.8.3, <0.9', +'configparser>=3.5.0, <3.6.0', 'croniter>=0.3.8, <0.4', 'dill>=0.2.2, <0.3', 'flask>=0.11, <0.12', @@ -211,7 +212,7 @@ def do_setup(): 'flask-swagger==0.2.13', 'flask-wtf==0.12', 'funcsigs==1.0.0', -'future>=0.15.0, <0.17', +'future>=0.16.0, <0.17', 'gitpython>=2.0.2', 'gunicorn>=19.3.0, <19.4.0', # 19.4.? seemed to have issues 'jinja2>=2.7.3, <2.9.0',
[jira] [Commented] (AIRFLOW-956) Setup readthedocs for versioned docs
[ https://issues.apache.org/jira/browse/AIRFLOW-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902490#comment-15902490 ] ASF subversion and git services commented on AIRFLOW-956: - Commit d7f8334d246bd3c6aa374fdfdb46fb59bb6bb626 in incubator-airflow's branch refs/heads/master from [~maxime.beauche...@apache.org] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d7f8334 ] Merge pull request #2133 from dhuang/AIRFLOW-956 > Setup readthedocs for versioned docs > > > Key: AIRFLOW-956 > URL: https://issues.apache.org/jira/browse/AIRFLOW-956 > Project: Apache Airflow > Issue Type: Improvement > Components: docs >Reporter: Daniel Huang >Assignee: Daniel Huang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-956) Setup readthedocs for versioned docs
[ https://issues.apache.org/jira/browse/AIRFLOW-956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902489#comment-15902489 ] ASF subversion and git services commented on AIRFLOW-956: - Commit f4be54f2535ca13819e20e742a8ed6ae522d1a17 in incubator-airflow's branch refs/heads/master from [~dxhuang] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=f4be54f ] [AIRFLOW-956] Get docs working on readthedocs.org > Setup readthedocs for versioned docs > > > Key: AIRFLOW-956 > URL: https://issues.apache.org/jira/browse/AIRFLOW-956 > Project: Apache Airflow > Issue Type: Improvement > Components: docs >Reporter: Daniel Huang >Assignee: Daniel Huang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[2/2] incubator-airflow git commit: Merge pull request #2133 from dhuang/AIRFLOW-956
Merge pull request #2133 from dhuang/AIRFLOW-956 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d7f8334d Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d7f8334d Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d7f8334d Branch: refs/heads/master Commit: d7f8334d246bd3c6aa374fdfdb46fb59bb6bb626 Parents: e79dee8 f4be54f Author: Maxime Beauchemin Authored: Wed Mar 8 20:56:41 2017 -0800 Committer: Maxime Beauchemin Committed: Wed Mar 8 20:56:41 2017 -0800 -- .readthedocs.yml | 20 1 file changed, 20 insertions(+) --
[1/2] incubator-airflow git commit: [AIRFLOW-956] Get docs working on readthedocs.org
Repository: incubator-airflow Updated Branches: refs/heads/master e79dee871 -> d7f8334d2 [AIRFLOW-956] Get docs working on readthedocs.org Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f4be54f2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f4be54f2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f4be54f2 Branch: refs/heads/master Commit: f4be54f2535ca13819e20e742a8ed6ae522d1a17 Parents: e79dee8 Author: Daniel Huang Authored: Wed Mar 8 19:42:29 2017 -0800 Committer: Daniel Huang Committed: Wed Mar 8 19:42:29 2017 -0800 -- .readthedocs.yml | 20 1 file changed, 20 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f4be54f2/.readthedocs.yml -- diff --git a/.readthedocs.yml b/.readthedocs.yml new file mode 100644 index 000..ca5b56d --- /dev/null +++ b/.readthedocs.yml @@ -0,0 +1,20 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +python: +pip_install: true +extra_requirements: +- doc +- docker +- gcp_api +- emr
[jira] [Created] (AIRFLOW-956) Setup readthedocs for versioned docs
Daniel Huang created AIRFLOW-956: Summary: Setup readthedocs for versioned docs Key: AIRFLOW-956 URL: https://issues.apache.org/jira/browse/AIRFLOW-956 Project: Apache Airflow Issue Type: Improvement Components: docs Reporter: Daniel Huang Assignee: Daniel Huang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-949) kill_process_tree does not kill the root process
[ https://issues.apache.org/jira/browse/AIRFLOW-949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Cederstrand updated AIRFLOW-949: - Attachment: helpers.patch Updated patch with better log messages > kill_process_tree does not kill the root process > > > Key: AIRFLOW-949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-949 > Project: Apache Airflow > Issue Type: Bug > Components: utils >Affects Versions: 1.8.0rc4 >Reporter: Erik Cederstrand > Labels: patch > Attachments: helpers.patch > > > The kill_process_tree() function in airflow/utils/helper.py does not attempt > to kill the root process. Since there's also a kill_descendant_processes() > function, I assume that was the intent. > Also, according to the comments, the intent is to send first SIGTERM, and > then SIGKILL, to decendant processes. But in fact, SIGTERM is sent twice. > The attached patch fixes both problems. > This was found while investigating why the airflow worker would not kill > certain jobs that had crashed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (AIRFLOW-949) kill_process_tree does not kill the root process
[ https://issues.apache.org/jira/browse/AIRFLOW-949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Cederstrand updated AIRFLOW-949: - Attachment: (was: helpers.patch) > kill_process_tree does not kill the root process > > > Key: AIRFLOW-949 > URL: https://issues.apache.org/jira/browse/AIRFLOW-949 > Project: Apache Airflow > Issue Type: Bug > Components: utils >Affects Versions: 1.8.0rc4 >Reporter: Erik Cederstrand > Labels: patch > > The kill_process_tree() function in airflow/utils/helper.py does not attempt > to kill the root process. Since there's also a kill_descendant_processes() > function, I assume that was the intent. > Also, according to the comments, the intent is to send first SIGTERM, and > then SIGKILL, to decendant processes. But in fact, SIGTERM is sent twice. > The attached patch fixes both problems. > This was found while investigating why the airflow worker would not kill > certain jobs that had crashed. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-935) Impossible to use plugin executors
[ https://issues.apache.org/jira/browse/AIRFLOW-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900919#comment-15900919 ] Stijn Verhaegen commented on AIRFLOW-935: - fix: https://github.com/apache/incubator-airflow/pull/2120#issuecomment-283656054 > Impossible to use plugin executors > -- > > Key: AIRFLOW-935 > URL: https://issues.apache.org/jira/browse/AIRFLOW-935 > Project: Apache Airflow > Issue Type: Bug > Components: executor, plugins >Affects Versions: Airflow 1.8, 1.9.0 >Reporter: Stijn Verhaegen > > revision eb5982d (included in 1.8) breaks plugin executors altogether. It > makes a new module for every plugin, so import statements need to be adapted, > but the executor selection is left unchanged, so it ends up assigning the > plugin module as an executor. > To reproduce: take any plugin which defines a custom executor and try to get > it loaded by setting `executor` in the airflow.cfg -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-860) Circular module dependency prevents loading of custom executor plugin
[
https://issues.apache.org/jira/browse/AIRFLOW-860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15900920#comment-15900920
]
Stijn Verhaegen commented on AIRFLOW-860:
-
fixed by: https://github.com/apache/incubator-airflow/pull/2120
> Circular module dependency prevents loading of custom executor plugin
> -
>
> Key: AIRFLOW-860
> URL: https://issues.apache.org/jira/browse/AIRFLOW-860
> Project: Apache Airflow
> Issue Type: Bug
> Components: executor, plugins
>Affects Versions: Airflow 1.8
> Environment: Linux RHEL 7.6, Python 2.7.13
>Reporter: Nick Ivanov
>
> Custom plugins cannot be loaded, which prevents airflow from running, due to
> apparent cyclic dependency in plugins_manager called in
> {{executors.\_\_init\_\_}} -- the top-level {{\_\_init\_\_}} attempts to load
> the default executor, which then goes back to plugins_manager etc.
> {noformat}
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/EGG-INFO/scripts/airflow",
> line 17, in
> from airflow import configuration
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
> line 31, in
> from airflow.models import DAG
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
> line 62, in
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
> line 55, in
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor TestExecutor not supported.
> {noformat}
> While attempting {{from airflow.plugins_manager import executors_modules}}
> cycles right back where it came from:
> {noformat}
> >>> from airflow.plugins_manager import executor_modules
> Traceback (most recent call last):
> File "", line 1, in
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/__init__.py",
> line 31, in
> from airflow.models import DAG
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/models.py",
> line 62, in
> from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor
> File
> "/usr/lib/python2.7/site-packages/airflow-1.9.0dev0_apache.incubating-py2.7.egg/airflow/executors/__init__.py",
> line 55, in
> raise AirflowException("Executor {0} not supported.".format(_EXECUTOR))
> airflow.exceptions.AirflowException: Executor LSFExecutor not supported.
> {noformat}
> To reproduce:
> # Create a custom executor, e.g. by copying {{local_executor.py}} into
> {{$AIRFLOW_HOME/plugins/test_executor.py}} and replacing occurrences of
> "Local" with "Test" in the code, which will produce {{TestExecutor}}
> # Update {{$AIRFLOW_HOME/airflow.cfg}} by setting in the {{\[core\]}}
> section: {{executor = TestExecutor}}
> # Run any airflow command, e.g. "airflow version"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
