[jira] [Resolved] (AIRFLOW-449) Add Whistle Labs as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-449. - Resolution: Fixed Issue resolved by pull request #1751 [https://github.com/apache/incubator-airflow/pull/1751] > Add Whistle Labs as an Airflow user > --- > > Key: AIRFLOW-449 > URL: https://issues.apache.org/jira/browse/AIRFLOW-449 > Project: Apache Airflow > Issue Type: Task >Reporter: Ananya Mishra >Assignee: Ananya Mishra >Priority: Trivial > > Add Whistle Labs as an Airflow user to the readme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-449) Add Whistle Labs as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429048#comment-15429048 ] ASF subversion and git services commented on AIRFLOW-449: - Commit 3cba29e020132cae924b0821a36ce0712fa3e2d6 in incubator-airflow's branch refs/heads/master from [~ananya77041] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3cba29e ] [AIRFLOW-449] Add Whistle Labs as an Airflow user Adding Whistle Labs as a user of Airflow. Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-449 Testing Done: - N/A Closes #1751 from ananya77041/master > Add Whistle Labs as an Airflow user > --- > > Key: AIRFLOW-449 > URL: https://issues.apache.org/jira/browse/AIRFLOW-449 > Project: Apache Airflow > Issue Type: Task >Reporter: Ananya Mishra >Assignee: Ananya Mishra >Priority: Trivial > > Add Whistle Labs as an Airflow user to the readme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-449) Add Whistle Labs as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429049#comment-15429049 ] ASF subversion and git services commented on AIRFLOW-449: - Commit 3cba29e020132cae924b0821a36ce0712fa3e2d6 in incubator-airflow's branch refs/heads/master from [~ananya77041] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=3cba29e ] [AIRFLOW-449] Add Whistle Labs as an Airflow user Adding Whistle Labs as a user of Airflow. Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-449 Testing Done: - N/A Closes #1751 from ananya77041/master > Add Whistle Labs as an Airflow user > --- > > Key: AIRFLOW-449 > URL: https://issues.apache.org/jira/browse/AIRFLOW-449 > Project: Apache Airflow > Issue Type: Task >Reporter: Ananya Mishra >Assignee: Ananya Mishra >Priority: Trivial > > Add Whistle Labs as an Airflow user to the readme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-449] Add Whistle Labs as an Airflow user
Repository: incubator-airflow Updated Branches: refs/heads/master df848a556 -> 3cba29e02 [AIRFLOW-449] Add Whistle Labs as an Airflow user Adding Whistle Labs as a user of Airflow. Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-449 Testing Done: - N/A Closes #1751 from ananya77041/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3cba29e0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3cba29e0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3cba29e0 Branch: refs/heads/master Commit: 3cba29e020132cae924b0821a36ce0712fa3e2d6 Parents: df848a5 Author: Ananya Mishra Authored: Fri Aug 19 16:38:37 2016 -0700 Committer: Siddharth Anand Committed: Fri Aug 19 16:38:37 2016 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3cba29e0/README.md -- diff --git a/README.md b/README.md index 2fb85a1..3c55e29 100644 --- a/README.md +++ b/README.md @@ -119,6 +119,7 @@ Currently **officially** using Airflow: 1. [Thumbtack](https://www.thumbtack.com/) [[@natekupp](https://github.com/natekupp)] 1. [WePay](http://www.wepay.com) [[@criccomini](https://github.com/criccomini) & [@mtagle](https://github.com/mtagle)] 1. [WeTransfer](https://github.com/WeTransfer) [[@jochem](https://github.com/jochem)] +1. [Whistle Labs](http://www.whistle.com) [[@ananya77041](https://github.com/ananya77041)] 1. Wooga 1. Xoom [[@gepser](https://github.com/gepser) & [@omarvides](https://github.com/omarvides)] 1. Yahoo!
[jira] [Created] (AIRFLOW-449) Add Whistle Labs as an Airflow user
Ananya Mishra created AIRFLOW-449: - Summary: Add Whistle Labs as an Airflow user Key: AIRFLOW-449 URL: https://issues.apache.org/jira/browse/AIRFLOW-449 Project: Apache Airflow Issue Type: Task Reporter: Ananya Mishra Assignee: Ananya Mishra Priority: Trivial Add Whistle Labs as an Airflow user to the readme. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-444) Add support for Google Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-444. - Resolution: Fixed Issue resolved by pull request #1747 [https://github.com/apache/incubator-airflow/pull/1747] > Add support for Google Authentication > - > > Key: AIRFLOW-444 > URL: https://issues.apache.org/jira/browse/AIRFLOW-444 > Project: Apache Airflow > Issue Type: New Feature > Components: security >Reporter: Ananya Mishra >Assignee: Ananya Mishra >Priority: Minor > > Support using Google OAuth 2.0 authentication to log into Airflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-444) Add support for Google Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429020#comment-15429020 ] ASF subversion and git services commented on AIRFLOW-444: - Commit df848a5564ed4b2d3281df79f77c51559e95c1a5 in incubator-airflow's branch refs/heads/master from [~ananya77041] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=df848a5 ] [AIRFLOW-444] Add Google authentication backend Add Google authentication backend. Add Google authentication information to security docs. Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-444 Testing Done: - Tested Google authentication backend locally with no issues This is mostly an adaptation of the GHE authentication backend. Closes #1747 from ananya77041/google_auth_backend > Add support for Google Authentication > - > > Key: AIRFLOW-444 > URL: https://issues.apache.org/jira/browse/AIRFLOW-444 > Project: Apache Airflow > Issue Type: New Feature > Components: security >Reporter: Ananya Mishra >Assignee: Ananya Mishra >Priority: Minor > > Support using Google OAuth 2.0 authentication to log into Airflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-444] Add Google authentication backend
Repository: incubator-airflow
Updated Branches:
refs/heads/master 7a5e1d832 -> df848a556
[AIRFLOW-444] Add Google authentication backend
Add Google authentication backend.
Add Google authentication information to security
docs.
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-444
Testing Done:
- Tested Google authentication backend locally
with no issues
This is mostly an adaptation of the GHE
authentication backend.
Closes #1747 from ananya77041/google_auth_backend
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/df848a55
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/df848a55
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/df848a55
Branch: refs/heads/master
Commit: df848a5564ed4b2d3281df79f77c51559e95c1a5
Parents: 7a5e1d8
Author: Ananya Mishra
Authored: Fri Aug 19 16:12:58 2016 -0700
Committer: Siddharth Anand
Committed: Fri Aug 19 16:12:58 2016 -0700
--
airflow/contrib/auth/backends/google_auth.py | 191 ++
docs/security.rst| 33
2 files changed, 224 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/df848a55/airflow/contrib/auth/backends/google_auth.py
--
diff --git a/airflow/contrib/auth/backends/google_auth.py
b/airflow/contrib/auth/backends/google_auth.py
new file mode 100644
index 000..8aed03a
--- /dev/null
+++ b/airflow/contrib/auth/backends/google_auth.py
@@ -0,0 +1,191 @@
+# Copyright 2016 Ananya Mishra (am...@cornell.edu)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+
+import flask_login
+
+# Need to expose these downstream
+# pylint: disable=unused-import
+from flask_login import (current_user,
+ logout_user,
+ login_required,
+ login_user)
+# pylint: enable=unused-import
+
+from flask import url_for, redirect, request
+
+from flask_oauthlib.client import OAuth
+
+from airflow import models, configuration, settings
+from airflow.configuration import AirflowConfigException
+
+_log = logging.getLogger(__name__)
+
+
+def get_config_param(param):
+return str(configuration.get('google', param))
+
+
+class GoogleUser(models.User):
+
+def __init__(self, user):
+self.user = user
+
+def is_active(self):
+'''Required by flask_login'''
+return True
+
+def is_authenticated(self):
+'''Required by flask_login'''
+return True
+
+def is_anonymous(self):
+'''Required by flask_login'''
+return False
+
+def get_id(self):
+'''Returns the current user id as required by flask_login'''
+return self.user.get_id()
+
+def data_profiling(self):
+'''Provides access to data profiling tools'''
+return True
+
+def is_superuser(self):
+'''Access all the things'''
+return True
+
+
+class AuthenticationError(Exception):
+pass
+
+
+class GoogleAuthBackend(object):
+
+def __init__(self):
+# self.google_host = get_config_param('host')
+self.login_manager = flask_login.LoginManager()
+self.login_manager.login_view = 'airflow.login'
+self.flask_app = None
+self.google_oauth = None
+self.api_rev = None
+
+def init_app(self, flask_app):
+self.flask_app = flask_app
+
+self.login_manager.init_app(self.flask_app)
+
+self.google_oauth = OAuth(self.flask_app).remote_app(
+'google',
+consumer_key=get_config_param('client_id'),
+consumer_secret=get_config_param('client_secret'),
+request_token_params={'scope':
'''https://www.googleapis.com/auth/userinfo.profile
+
https://www.googleapis.com/auth/userinfo.email'''},
+base_url='https://www.google.com/accounts/',
+request_token_url=None,
+access_token_method='POST',
+access_token_url='https://accounts.google.com/o/oauth2/token',
+authorize_url='https://accounts.google.com/o/oauth2/auth')
+
+self.login_manager.user_lo
[jira] [Commented] (AIRFLOW-444) Add support for Google Authentication
[ https://issues.apache.org/jira/browse/AIRFLOW-444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429019#comment-15429019 ] ASF subversion and git services commented on AIRFLOW-444: - Commit df848a5564ed4b2d3281df79f77c51559e95c1a5 in incubator-airflow's branch refs/heads/master from [~ananya77041] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=df848a5 ] [AIRFLOW-444] Add Google authentication backend Add Google authentication backend. Add Google authentication information to security docs. Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-444 Testing Done: - Tested Google authentication backend locally with no issues This is mostly an adaptation of the GHE authentication backend. Closes #1747 from ananya77041/google_auth_backend > Add support for Google Authentication > - > > Key: AIRFLOW-444 > URL: https://issues.apache.org/jira/browse/AIRFLOW-444 > Project: Apache Airflow > Issue Type: New Feature > Components: security >Reporter: Ananya Mishra >Assignee: Ananya Mishra >Priority: Minor > > Support using Google OAuth 2.0 authentication to log into Airflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-445) Missing DataProc operator submit options
[ https://issues.apache.org/jira/browse/AIRFLOW-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429005#comment-15429005 ] ASF subversion and git services commented on AIRFLOW-445: - Commit 7a5e1d832376af5e048a20ab718c41a20ff6cdf8 in incubator-airflow's branch refs/heads/master from [~cerisier] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7a5e1d8 ] [AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options Adds support equivalent support for --files and --py-files cli options. Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-445 - Missing dataproc operator submit options](https://issues.apache.org/jira/browse/AIR FLOW-445) Changes Done: - Adds support equivalent support for --files and --py-files cli options. Testing Done: - Non existent for that operator yet, tested manually by submitting a job with extra files and py-files Signed-off-by: Corentin Kerisit Closes #1748 from cerisier/dataproc_additional_options > Missing DataProc operator submit options > > > Key: AIRFLOW-445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-445 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, gcp, hooks, operators >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Minor > Labels: contrib, dataproc, spark > > Options to upload files and py-files are missing from the current dataproc > operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-445) Missing DataProc operator submit options
[ https://issues.apache.org/jira/browse/AIRFLOW-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-445. - Resolution: Fixed Issue resolved by pull request #1748 [https://github.com/apache/incubator-airflow/pull/1748] > Missing DataProc operator submit options > > > Key: AIRFLOW-445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-445 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, gcp, hooks, operators >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Minor > Labels: contrib, dataproc, spark > > Options to upload files and py-files are missing from the current dataproc > operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-445) Missing DataProc operator submit options
[ https://issues.apache.org/jira/browse/AIRFLOW-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429006#comment-15429006 ] ASF subversion and git services commented on AIRFLOW-445: - Commit 7a5e1d832376af5e048a20ab718c41a20ff6cdf8 in incubator-airflow's branch refs/heads/master from [~cerisier] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7a5e1d8 ] [AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options Adds support equivalent support for --files and --py-files cli options. Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-445 - Missing dataproc operator submit options](https://issues.apache.org/jira/browse/AIR FLOW-445) Changes Done: - Adds support equivalent support for --files and --py-files cli options. Testing Done: - Non existent for that operator yet, tested manually by submitting a job with extra files and py-files Signed-off-by: Corentin Kerisit Closes #1748 from cerisier/dataproc_additional_options > Missing DataProc operator submit options > > > Key: AIRFLOW-445 > URL: https://issues.apache.org/jira/browse/AIRFLOW-445 > Project: Apache Airflow > Issue Type: Improvement > Components: contrib, gcp, hooks, operators >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Minor > Labels: contrib, dataproc, spark > > Options to upload files and py-files are missing from the current dataproc > operator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-446) Add Zenly as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15429004#comment-15429004 ] ASF subversion and git services commented on AIRFLOW-446: - Commit 7a5e1d832376af5e048a20ab718c41a20ff6cdf8 in incubator-airflow's branch refs/heads/master from [~cerisier] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=7a5e1d8 ] [AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options Adds support equivalent support for --files and --py-files cli options. Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-445 - Missing dataproc operator submit options](https://issues.apache.org/jira/browse/AIR FLOW-445) Changes Done: - Adds support equivalent support for --files and --py-files cli options. Testing Done: - Non existent for that operator yet, tested manually by submitting a job with extra files and py-files Signed-off-by: Corentin Kerisit Closes #1748 from cerisier/dataproc_additional_options > Add Zenly as an Airflow user > > > Key: AIRFLOW-446 > URL: https://issues.apache.org/jira/browse/AIRFLOW-446 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Trivial > > Add Zenly as an Airflow user to the Readme, following [the call to reference > from Twitter|https://twitter.com/ApacheAirflow/status/763618344255692805]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options
Repository: incubator-airflow
Updated Branches:
refs/heads/master d20299177 -> 7a5e1d832
[AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options
Adds support equivalent support for --files and
--py-files cli options.
Signed-off-by: Corentin Kerisit
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
- [AIRFLOW-445 - Missing dataproc operator submit
options](https://issues.apache.org/jira/browse/AIR
FLOW-445)
Changes Done:
- Adds support equivalent support for --files and
--py-files cli options.
Testing Done:
- Non existent for that operator yet, tested
manually by submitting a job with extra files and
py-files
Signed-off-by: Corentin Kerisit
Closes #1748 from
cerisier/dataproc_additional_options
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7a5e1d83
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7a5e1d83
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7a5e1d83
Branch: refs/heads/master
Commit: 7a5e1d832376af5e048a20ab718c41a20ff6cdf8
Parents: d202991
Author: Corentin Kerisit
Authored: Fri Aug 19 16:02:35 2016 -0700
Committer: Siddharth Anand
Committed: Fri Aug 19 16:02:35 2016 -0700
--
airflow/contrib/hooks/gcp_dataproc_hook.py | 8
airflow/contrib/operators/dataproc_operator.py | 21 +
2 files changed, 29 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7a5e1d83/airflow/contrib/hooks/gcp_dataproc_hook.py
--
diff --git a/airflow/contrib/hooks/gcp_dataproc_hook.py
b/airflow/contrib/hooks/gcp_dataproc_hook.py
index 3dd1f64..fab71c5 100644
--- a/airflow/contrib/hooks/gcp_dataproc_hook.py
+++ b/airflow/contrib/hooks/gcp_dataproc_hook.py
@@ -107,6 +107,14 @@ class _DataProcJobBuilder:
if archives is not None:
self.job["job"][self.job_type]["archiveUris"] = archives
+def add_file_uris(self, files):
+if files is not None:
+self.job["job"][self.job_type]["fileUris"] = files
+
+def add_python_file_uris(self, pyfiles):
+if pyfiles is not None:
+self.job["job"][self.job_type]["pythonFileUris"] = pyfiles
+
def set_main(self, main_jar, main_class):
if main_class is not None and main_jar is not None:
raise Exception("Set either main_jar or main_class")
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7a5e1d83/airflow/contrib/operators/dataproc_operator.py
--
diff --git a/airflow/contrib/operators/dataproc_operator.py
b/airflow/contrib/operators/dataproc_operator.py
index 4d3e1e7..33e5f79 100644
--- a/airflow/contrib/operators/dataproc_operator.py
+++ b/airflow/contrib/operators/dataproc_operator.py
@@ -256,6 +256,7 @@ class DataProcSparkOperator(BaseOperator):
main_class=None,
arguments=None,
archives=None,
+files=None,
dataproc_cluster='cluster-1',
dataproc_spark_properties=None,
dataproc_spark_jars=None,
@@ -277,6 +278,8 @@ class DataProcSparkOperator(BaseOperator):
:param archives: List of archived files that will be unpacked in the
work
directory. Should be stored in Cloud Storage.
:type archives: list
+:param files: List of files to be copied to the working directory
+:type files: list
:param dataproc_cluster: The id of the DataProc cluster.
:type dataproc_cluster: string
:param dataproc_spark_properties: Map for the Pig properties. Ideal to
put in
@@ -299,6 +302,7 @@ class DataProcSparkOperator(BaseOperator):
self.main_class = main_class
self.arguments = arguments
self.archives = archives
+self.files = files
self.dataproc_cluster = dataproc_cluster
self.dataproc_properties = dataproc_spark_properties
self.dataproc_jars = dataproc_spark_jars
@@ -313,6 +317,7 @@ class DataProcSparkOperator(BaseOperator):
job.add_args(self.arguments)
job.add_jar_file_uris(self.dataproc_jars)
job.add_archive_uris(self.archives)
+job.add_file_uris(self.files)
hook.submit(hook.project_id, job.build())
@@ -332,6 +337,7 @@ class DataProcHadoopOperator(BaseOperator):
main_class=None,
arguments=None,
archives=None,
+files=None,
dataproc_cluster='cluster-1',
dataproc_hadoop_properties=None,
dataproc_hadoop_jars=None,
@@ -353,6 +359,8 @@ class DataProcHadoopOperator(BaseOperator):
:param archives: List of archived files that will be
[jira] [Resolved] (AIRFLOW-448) Add Apigee as an official user
[ https://issues.apache.org/jira/browse/AIRFLOW-448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-448. - Resolution: Fixed Issue resolved by pull request #1750 [https://github.com/apache/incubator-airflow/pull/1750] > Add Apigee as an official user > -- > > Key: AIRFLOW-448 > URL: https://issues.apache.org/jira/browse/AIRFLOW-448 > Project: Apache Airflow > Issue Type: Task >Reporter: Ben Tallman >Assignee: Ben Tallman >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-448) Add Apigee as an official user
[ https://issues.apache.org/jira/browse/AIRFLOW-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428917#comment-15428917 ] ASF subversion and git services commented on AIRFLOW-448: - Commit d20299177dc6266786b46dfe1a7675aa853840e3 in incubator-airflow's branch refs/heads/master from [~btall...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d202991 ] [AIRFLOW-448] Adding Apigee as an official user of Airflow Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-448] https://issues.apache.org/jira/browse/AIRFLOW-448 Closes #1750 from btallman/master > Add Apigee as an official user > -- > > Key: AIRFLOW-448 > URL: https://issues.apache.org/jira/browse/AIRFLOW-448 > Project: Apache Airflow > Issue Type: Task >Reporter: Ben Tallman >Assignee: Ben Tallman >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-448) Add Apigee as an official user
[ https://issues.apache.org/jira/browse/AIRFLOW-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428915#comment-15428915 ] ASF subversion and git services commented on AIRFLOW-448: - Commit d20299177dc6266786b46dfe1a7675aa853840e3 in incubator-airflow's branch refs/heads/master from [~btall...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d202991 ] [AIRFLOW-448] Adding Apigee as an official user of Airflow Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-448] https://issues.apache.org/jira/browse/AIRFLOW-448 Closes #1750 from btallman/master > Add Apigee as an official user > -- > > Key: AIRFLOW-448 > URL: https://issues.apache.org/jira/browse/AIRFLOW-448 > Project: Apache Airflow > Issue Type: Task >Reporter: Ben Tallman >Assignee: Ben Tallman >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-448) Add Apigee as an official user
[ https://issues.apache.org/jira/browse/AIRFLOW-448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428916#comment-15428916 ] ASF subversion and git services commented on AIRFLOW-448: - Commit d20299177dc6266786b46dfe1a7675aa853840e3 in incubator-airflow's branch refs/heads/master from [~btall...@gmail.com] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=d202991 ] [AIRFLOW-448] Adding Apigee as an official user of Airflow Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-448] https://issues.apache.org/jira/browse/AIRFLOW-448 Closes #1750 from btallman/master > Add Apigee as an official user > -- > > Key: AIRFLOW-448 > URL: https://issues.apache.org/jira/browse/AIRFLOW-448 > Project: Apache Airflow > Issue Type: Task >Reporter: Ben Tallman >Assignee: Ben Tallman >Priority: Trivial > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-448] Adding Apigee as an official user of Airflow
Repository: incubator-airflow Updated Branches: refs/heads/master 6af3a3dcd -> d20299177 [AIRFLOW-448] Adding Apigee as an official user of Airflow Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-448] https://issues.apache.org/jira/browse/AIRFLOW-448 Closes #1750 from btallman/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d2029917 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d2029917 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d2029917 Branch: refs/heads/master Commit: d20299177dc6266786b46dfe1a7675aa853840e3 Parents: 6af3a3d Author: Benjamin Tallman Authored: Fri Aug 19 15:04:45 2016 -0700 Committer: Siddharth Anand Committed: Fri Aug 19 15:04:52 2016 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d2029917/README.md -- diff --git a/README.md b/README.md index b213081..2fb85a1 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,7 @@ Currently **officially** using Airflow: 1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)] 1. [Agari](https://github.com/agaridata) [[@r39132](https://github.com/r39132)] 1. [allegro.pl](http://allegro.tech/) [[@kretes](https://github.com/kretes)] +1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)] 1. [Bellhops](https://github.com/bellhops) 1. BlueApron [[@jasonjho](https://github.com/jasonjho) & [@matthewdavidhauser](https://github.com/matthewdavidhauser)] 1. [Blue Yonder](http://www.blue-yonder.com) [[@blue-yonder](https://github.com/blue-yonder)]
[jira] [Resolved] (AIRFLOW-431) Add CLI for CRUD operations on pools
[ https://issues.apache.org/jira/browse/AIRFLOW-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-431. - Resolution: Fixed Issue resolved by pull request #1735 [https://github.com/apache/incubator-airflow/pull/1735] > Add CLI for CRUD operations on pools > > > Key: AIRFLOW-431 > URL: https://issues.apache.org/jira/browse/AIRFLOW-431 > Project: Apache Airflow > Issue Type: Bug >Reporter: Siddharth Anand >Assignee: Siddharth Anand > > Currently, we need to rely on the UI to create a pool. It would be great if > we could simply do this via the CLI so that we could leverage automation! > The new CLI command will support CRUD operations on pools via get, set, and > delete subcommands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-431) Add CLI for CRUD operations on pools
[ https://issues.apache.org/jira/browse/AIRFLOW-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428912#comment-15428912 ] ASF subversion and git services commented on AIRFLOW-431: - Commit 6af3a3dcdffc6678a42adda6cf0b986d802ea867 in incubator-airflow's branch refs/heads/master from Siddharth Anand [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6af3a3d ] [AIRFLOW-431] Add CLI for CRUD operations on pools Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-431 Testing Done: - Added unit tests Closes #1735 from r39132/master > Add CLI for CRUD operations on pools > > > Key: AIRFLOW-431 > URL: https://issues.apache.org/jira/browse/AIRFLOW-431 > Project: Apache Airflow > Issue Type: Bug >Reporter: Siddharth Anand >Assignee: Siddharth Anand > > Currently, we need to rely on the UI to create a pool. It would be great if > we could simply do this via the CLI so that we could leverage automation! > The new CLI command will support CRUD operations on pools via get, set, and > delete subcommands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-431) Add CLI for CRUD operations on pools
[ https://issues.apache.org/jira/browse/AIRFLOW-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428911#comment-15428911 ] ASF subversion and git services commented on AIRFLOW-431: - Commit 6af3a3dcdffc6678a42adda6cf0b986d802ea867 in incubator-airflow's branch refs/heads/master from Siddharth Anand [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6af3a3d ] [AIRFLOW-431] Add CLI for CRUD operations on pools Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - https://issues.apache.org/jira/browse/AIRFLOW-431 Testing Done: - Added unit tests Closes #1735 from r39132/master > Add CLI for CRUD operations on pools > > > Key: AIRFLOW-431 > URL: https://issues.apache.org/jira/browse/AIRFLOW-431 > Project: Apache Airflow > Issue Type: Bug >Reporter: Siddharth Anand >Assignee: Siddharth Anand > > Currently, we need to rely on the UI to create a pool. It would be great if > we could simply do this via the CLI so that we could leverage automation! > The new CLI command will support CRUD operations on pools via get, set, and > delete subcommands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-431] Add CLI for CRUD operations on pools
Repository: incubator-airflow
Updated Branches:
refs/heads/master 6d42d0c35 -> 6af3a3dcd
[AIRFLOW-431] Add CLI for CRUD operations on pools
Dear Airflow Maintainers,
Please accept this PR that addresses the following
issues:
-
https://issues.apache.org/jira/browse/AIRFLOW-431
Testing Done:
- Added unit tests
Closes #1735 from r39132/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6af3a3dc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6af3a3dc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6af3a3dc
Branch: refs/heads/master
Commit: 6af3a3dcdffc6678a42adda6cf0b986d802ea867
Parents: 6d42d0c
Author: Siddharth Anand
Authored: Fri Aug 19 15:01:01 2016 -0700
Committer: Siddharth Anand
Committed: Fri Aug 19 15:01:01 2016 -0700
--
airflow/bin/cli.py | 59 ++---
tests/core.py | 9
2 files changed, 65 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6af3a3dc/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 3ca608f..6e05d17 100755
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -42,7 +42,8 @@ from airflow import configuration as conf
from airflow.exceptions import AirflowException
from airflow.executors import DEFAULT_EXECUTOR
from airflow.models import (DagModel, DagBag, TaskInstance,
-DagPickle, DagRun, Variable, DagStat)
+DagPickle, DagRun, Variable, DagStat,
+Pool)
from airflow.utils import db as db_utils
from airflow.utils import logging as logging_utils
from airflow.utils.state import State
@@ -187,6 +188,39 @@ def trigger_dag(args):
logging.info("Created {}".format(trigger))
+def pool(args):
+session = settings.Session()
+if args.get or (args.set and args.set[0]) or args.delete:
+name = args.get or args.delete or args.set[0]
+pool = (
+session.query(Pool)
+.filter(Pool.pool == name)
+.first())
+if pool and args.get:
+print("{} ".format(pool))
+return
+elif not pool and (args.get or args.delete):
+print("No pool named {} found".format(name))
+elif not pool and args.set:
+pool = Pool(
+pool=name,
+slots=args.set[1],
+description=args.set[2])
+session.add(pool)
+session.commit()
+print("{} ".format(pool))
+elif pool and args.set:
+pool.slots = args.set[1]
+pool.description = args.set[2]
+session.commit()
+print("{} ".format(pool))
+return
+elif pool and args.delete:
+session.query(Pool).filter_by(pool=args.delete).delete()
+session.commit()
+print("Pool {} deleted".format(name))
+
+
def variables(args):
if args.get:
@@ -220,6 +254,7 @@ def variables(args):
msg = "\n".join(var.key for var in vars)
print(msg)
+
def import_helper(filepath):
with open(filepath, 'r') as varfile:
var = varfile.read()
@@ -242,6 +277,7 @@ def import_helper(filepath):
finally:
print("{} of {} variables successfully updated.".format(n, len(d)))
+
def export_helper(filepath):
session = settings.Session()
qry = session.query(Variable).all()
@@ -571,7 +607,6 @@ def restart_workers(gunicorn_master_proc,
num_workers_expected):
get_num_workers_running(gunicorn_master_proc))
while True:
-
num_workers_running = get_num_workers_running(gunicorn_master_proc)
num_ready_workers_running =
get_num_ready_workers_running(gunicorn_master_proc)
@@ -975,6 +1010,20 @@ class CLIFactory(object):
'conf': Arg(
('-c', '--conf'),
"JSON string that gets pickled into the DagRun's conf attribute"),
+# pool
+'pool_set': Arg(
+("-s", "--set"),
+nargs=3,
+metavar=('NAME', 'SLOT_COUNT', 'POOL_DESCRIPTION'),
+help="Set pool slot count and description, respectively"),
+'pool_get': Arg(
+("-g", "--get"),
+metavar='NAME',
+help="Get pool info"),
+'pool_delete': Arg(
+("-x", "--delete"),
+metavar="NAME",
+help="Delete a pool"),
# variables
'set': Arg(
("-s", "--set"),
@@ -1158,8 +1207,12 @@ class CLIFactory(object):
'help': "Trigger a DAG run",
'args': ('dag_id', 'subdir', 'run_id', 'conf'),
}, {
+'func': pool,
+'help': "CRUD operations on pools",
+"args": ('pool_set',
[jira] [Created] (AIRFLOW-448) Add Apigee as an official user
Ben Tallman created AIRFLOW-448: --- Summary: Add Apigee as an official user Key: AIRFLOW-448 URL: https://issues.apache.org/jira/browse/AIRFLOW-448 Project: Apache Airflow Issue Type: Task Reporter: Ben Tallman Assignee: Ben Tallman Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-434) max_dag_run_reached blocks dag state change and new task scheduling
[
https://issues.apache.org/jira/browse/AIRFLOW-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428184#comment-15428184
]
Michael DeWulf commented on AIRFLOW-434:
Unfortunately no, I don't have a fix. I was just reporting that I'm seeing the
same problem.
> max_dag_run_reached blocks dag state change and new task scheduling
> ---
>
> Key: AIRFLOW-434
> URL: https://issues.apache.org/jira/browse/AIRFLOW-434
> Project: Apache Airflow
> Issue Type: Bug
>Reporter: Li Xuanji
>Assignee: Siddharth Anand
>Priority: Blocker
>
> Using the following DAG:
> ```
> from airflow import DAG
> from airflow.operators.bash_operator import BashOperator
> from datetime import datetime, timedelta
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime(2016, 1, 1, 1, 0),
> 'email': ['xua...@gmail.com'],
> 'email_on_failure': True,
> 'email_on_retry': False,
> 'retries': 3,
> 'retry_delay': timedelta(minutes=1),
> }
> dag = DAG('bash_bash_bash', default_args=default_args,
> schedule_interval=timedelta(seconds=10))
> # t1, t2 and t3 are examples of tasks created by instatiating operators
> t1 = BashOperator(
> task_id='print_date',
> bash_command='date',
> dag=dag)
> t2 = BashOperator(
> task_id='sleep',
> bash_command='sleep 120',
> retries=3,
> dag=dag)
> templated_command = """
> {% for i in range(5) %}
> echo "{{ ds }}"
> echo "{{ macros.ds_add(ds, 7)}}"
> echo "{{ params.my_param }}"
> {% endfor %}
> """
> t3 = BashOperator(
> task_id='templated',
> bash_command=templated_command,
> params={'my_param': 'Parameter I passed in'},
> dag=dag)
> t2.set_upstream(t1)
> t3.set_upstream(t1)
> ```
> and an `airflow.cfg` that contains this:
> ```
> min_file_process_interval = 1
> ```
> The state eventually becomes this:
> http://imgur.com/a/5bRTe
> The scheduler should be marking the 14 leftmost dagruns as success, but does
> not. the scheduler should also be scheduling tasks for the last two dagruns.
> A look at the logs explains the probable cause:
> ```
> [2016-08-16 15:12:10,257] {jobs.py:1446} DagFileProcessor174 INFO -
> Processing file /Users/xuanji_li/airflow/dags/bash_bash_bash.py for tasks to
> queue
> [2016-08-16 15:12:10,258] {models.py:162} DagFileProcessor174 INFO - Filling
> up the DagBag from /Users/xuanji_li/airflow/dags/bash_bash_bash.py
> [2016-08-16 15:12:10,267] {jobs.py:1460} DagFileProcessor174 INFO - DAG(s)
> ['bash_bash_bash'] retrieved from
> /Users/xuanji_li/airflow/dags/bash_bash_bash.py
> [2016-08-16 15:12:10,289] {jobs.py:1062} DagFileProcessor174 INFO - Not
> processing DAG bash_bash_bash since its max runs has been reached
> [2016-08-16 15:12:10,290] {models.py:313} DagFileProcessor174 INFO - Finding
> 'running' jobs without a recent heartbeat
> [2016-08-16 15:12:10,290] {models.py:319} DagFileProcessor174 INFO - Failing
> jobs without heartbeat after 2016-08-16 15:09:55.290479
> ```
> It seems that processing of the dagrun is skipped completely because there
> are already 16 running dagruns.
> Binary search tracked down this commit as the one that introduced the bug.
> The logic added looks wrong to me.
> https://github.com/apache/incubator-airflow/pull/1716
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AIRFLOW-446) Add Zenly as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427731#comment-15427731 ] ASF subversion and git services commented on AIRFLOW-446: - Commit 6d42d0c35294e4864b6f2f424e20c8419fafbf73 in incubator-airflow's branch refs/heads/master from [~cerisier] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6d42d0c ] [AIRFLOW-446] Add Zenly as an airflow user Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-446 - Add Zenly as an Airflow user](htt ps://issues.apache.org/jira/browse/AIRFLOW-446) Testing Done: - Non Applicable Signed-off-by: Corentin Kerisit Closes #1749 from cerisier/new_user > Add Zenly as an Airflow user > > > Key: AIRFLOW-446 > URL: https://issues.apache.org/jira/browse/AIRFLOW-446 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Trivial > > Add Zenly as an Airflow user to the Readme, following [the call to reference > from Twitter|https://twitter.com/ApacheAirflow/status/763618344255692805]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (AIRFLOW-446) Add Zenly as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand resolved AIRFLOW-446. - Resolution: Fixed Issue resolved by pull request #1749 [https://github.com/apache/incubator-airflow/pull/1749] > Add Zenly as an Airflow user > > > Key: AIRFLOW-446 > URL: https://issues.apache.org/jira/browse/AIRFLOW-446 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Trivial > > Add Zenly as an Airflow user to the Readme, following [the call to reference > from Twitter|https://twitter.com/ApacheAirflow/status/763618344255692805]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-446) Add Zenly as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427732#comment-15427732 ] ASF subversion and git services commented on AIRFLOW-446: - Commit 6d42d0c35294e4864b6f2f424e20c8419fafbf73 in incubator-airflow's branch refs/heads/master from [~cerisier] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6d42d0c ] [AIRFLOW-446] Add Zenly as an airflow user Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-446 - Add Zenly as an Airflow user](htt ps://issues.apache.org/jira/browse/AIRFLOW-446) Testing Done: - Non Applicable Signed-off-by: Corentin Kerisit Closes #1749 from cerisier/new_user > Add Zenly as an Airflow user > > > Key: AIRFLOW-446 > URL: https://issues.apache.org/jira/browse/AIRFLOW-446 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Trivial > > Add Zenly as an Airflow user to the Readme, following [the call to reference > from Twitter|https://twitter.com/ApacheAirflow/status/763618344255692805]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AIRFLOW-446) Add Zenly as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427730#comment-15427730 ] ASF subversion and git services commented on AIRFLOW-446: - Commit 6d42d0c35294e4864b6f2f424e20c8419fafbf73 in incubator-airflow's branch refs/heads/master from [~cerisier] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=6d42d0c ] [AIRFLOW-446] Add Zenly as an airflow user Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-446 - Add Zenly as an Airflow user](htt ps://issues.apache.org/jira/browse/AIRFLOW-446) Testing Done: - Non Applicable Signed-off-by: Corentin Kerisit Closes #1749 from cerisier/new_user > Add Zenly as an Airflow user > > > Key: AIRFLOW-446 > URL: https://issues.apache.org/jira/browse/AIRFLOW-446 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Trivial > > Add Zenly as an Airflow user to the Readme, following [the call to reference > from Twitter|https://twitter.com/ApacheAirflow/status/763618344255692805]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
incubator-airflow git commit: [AIRFLOW-446] Add Zenly as an airflow user
Repository: incubator-airflow Updated Branches: refs/heads/master 38972162c -> 6d42d0c35 [AIRFLOW-446] Add Zenly as an airflow user Signed-off-by: Corentin Kerisit Dear Airflow Maintainers, Please accept this PR that addresses the following issues: - [AIRFLOW-446 - Add Zenly as an Airflow user](htt ps://issues.apache.org/jira/browse/AIRFLOW-446) Testing Done: - Non Applicable Signed-off-by: Corentin Kerisit Closes #1749 from cerisier/new_user Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6d42d0c3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6d42d0c3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6d42d0c3 Branch: refs/heads/master Commit: 6d42d0c35294e4864b6f2f424e20c8419fafbf73 Parents: 3897216 Author: Corentin Kerisit Authored: Fri Aug 19 00:10:57 2016 -0700 Committer: Siddharth Anand Committed: Fri Aug 19 00:11:08 2016 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6d42d0c3/README.md -- diff --git a/README.md b/README.md index c5c27d1..b213081 100644 --- a/README.md +++ b/README.md @@ -123,6 +123,7 @@ Currently **officially** using Airflow: 1. Yahoo! 1. [Zapier](https://www.zapier.com) [[@drknexus](https://github.com/drknexus) & [@statwonk](https://github.com/statwonk)] 1. [Zendesk](https://www.github.com/zendesk) +1. [Zenly](https://zen.ly) [[@cerisier](https://github.com/cerisier) & [@jbdalido](https://github.com/jbdalido)] ## Links
[jira] [Updated] (AIRFLOW-446) Add Zenly as an Airflow user
[ https://issues.apache.org/jira/browse/AIRFLOW-446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Anand updated AIRFLOW-446: External issue URL: https://github.com/apache/incubator-airflow/pull/1749 > Add Zenly as an Airflow user > > > Key: AIRFLOW-446 > URL: https://issues.apache.org/jira/browse/AIRFLOW-446 > Project: Apache Airflow > Issue Type: Task > Components: docs >Reporter: Corentin Kerisit >Assignee: Corentin Kerisit >Priority: Trivial > > Add Zenly as an Airflow user to the Readme, following [the call to reference > from Twitter|https://twitter.com/ApacheAirflow/status/763618344255692805]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (AIRFLOW-447) Python 3 map object cannot be json-serialized - need lists instead
Georg Walther created AIRFLOW-447: - Summary: Python 3 map object cannot be json-serialized - need lists instead Key: AIRFLOW-447 URL: https://issues.apache.org/jira/browse/AIRFLOW-447 Project: Apache Airflow Issue Type: Bug Components: contrib Affects Versions: Airflow 1.7.1.3 Environment: Python 3 Reporter: Georg Walther Priority: Blocker On this line, `source_uris` are generated with `map`: https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/gcs_to_bq.py#L124 In Python 2 `map` would return a list, however in Python 3 `map` returns a `map` object / generator. The Python 3 `map` object cannot be json-serialized downstream when storing corresponding job objects in the database: https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/bigquery_hook.py#L441 Constructing `source_uris` with a list comprehension instead of `map` resolves the issue for Python 3 and retains compatibility with Python 2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
