[incubator-airflow-ci] branch master updated: minor change
This is an automated email from the ASF dual-hosted git repository. sanand pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-airflow-ci.git The following commit(s) were added to refs/heads/master by this push: new bd5663d minor change bd5663d is described below commit bd5663dfece15eb1909cc04d1147a0e2ca028a2c Author: Sid Anand AuthorDate: Wed Sep 5 18:40:52 2018 -0700 minor change --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 2d302fa..b5848ee 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ **Do not use this image for production** -The main purpose of this repository is to provide a number of Docker containers, being build by the Docker Hub service: https://hub.docker.com/r/airflowci/incubator-airflow-ci/builds/ +The main purpose of this repository is to provide a number of Docker containers, being built by the Docker Hub service: https://hub.docker.com/r/airflowci/incubator-airflow-ci/builds/ The container contains a couple of the dependencies that are used for running tests against Apache Airflow.
incubator-airflow git commit: [AIRFLOW-XXX] Add Strongmind to the Readme
Repository: incubator-airflow Updated Branches: refs/heads/master a338f3276 -> 096ba9ecd [AIRFLOW-XXX] Add Strongmind to the Readme Closes #3655 from tomchapin/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/096ba9ec Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/096ba9ec Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/096ba9ec Branch: refs/heads/master Commit: 096ba9ecd961cdaebd062599f408571ffb21165a Parents: a338f32 Author: Tom Chapin Authored: Fri Jul 27 12:39:24 2018 -0700 Committer: r39132 Committed: Fri Jul 27 12:39:29 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/096ba9ec/README.md -- diff --git a/README.md b/README.md index f5a5133..7256565 100644 --- a/README.md +++ b/README.md @@ -231,6 +231,7 @@ Currently **officially** using Airflow: 1. [Spotify](https://github.com/spotify) [[@znichols](https://github.com/znichols)] 1. [Stackspace](https://beta.stackspace.io/) 1. [Stripe](https://stripe.com) [[@jbalogh](https://github.com/jbalogh)] +1. [Strongmind](https://www.strongmind.com) [[@tomchapin](https://github.com/tomchapin) & [@wongstein](https://github.com/wongstein)] 1. [Tails.com](https://tails.com/) [[@alanmcruickshank](https://github.com/alanmcruickshank)] 1. [Thinking Machines](https://thinkingmachin.es) [[@marksteve](https://github.com/marksteve)] 1. [Thinknear](https://www.thinknear.com/) [[@d3cay1](https://github.com/d3cay1), [@ccson](https://github.com/ccson), & [@ababian](https://github.com/ababian)]
incubator-airflow git commit: [AIRFLOW-XXX] Add Vidio and BBM to README
Repository: incubator-airflow Updated Branches: refs/heads/master 53e021cde -> 8715bc45b [AIRFLOW-XXX] Add Vidio and BBM to README Closes #3631 from BagusThanatos/add-kmk-bbm Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8715bc45 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8715bc45 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8715bc45 Branch: refs/heads/master Commit: 8715bc45b448c9c34684b30bf2422b04a4ea1d8f Parents: 53e021c Author: Bagus Nurtomo Authored: Wed Jul 25 19:13:46 2018 -0700 Committer: r39132 Committed: Wed Jul 25 19:13:50 2018 -0700 -- README.md | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8715bc45/README.md -- diff --git a/README.md b/README.md index e1de8ff..0302495 100644 --- a/README.md +++ b/README.md @@ -100,6 +100,7 @@ Currently **officially** using Airflow: 1. [BalanceHero](http://truebalance.io/) [[@swalloow](https://github.com/swalloow)] 1. [Banco de Formaturas](https://www.bancodeformaturas.com.br) [[@guiligan](https://github.com/guiligan)] 1. [BandwidthX](http://www.bandwidthx.com) [[@dineshdsharma](https://github.com/dineshdsharma)] +1. [BBM](https://www.bbm.com/) 1. [Bellhops](https://github.com/bellhops) 1. [BelugaDB](https://belugadb.com) [[@fabio-nukui](https://github.com/fabio-nukui) & [@joao-sallaberry](http://github.com/joao-sallaberry) & [@lucianoviola](https://github.com/lucianoviola) & [@tmatuki](https://github.com/tmatuki)] 1. [BlaBlaCar](https://www.blablacar.com) [[@puckel](https://github.com/puckel) & [@wmorin](https://github.com/wmorin)] @@ -243,6 +244,7 @@ Currently **officially** using Airflow: 1. [Upsight](https://www.upsight.com) [[@dhuang](https://github.com/dhuang)] 1. [Vente-Exclusive.com](http://www.vente-exclusive.com/) [[@alexvanboxel](https://github.com/alexvanboxel)] 1. [Vevo](https://www.vevo.com/) [[@csetiawan](https://github.com/csetiawan) & [@jerrygillespie](https://github.com/jerrygillespie)] +1. [Vidio](https://www.vidio.com/) 1. [Vnomics](https://github.com/vnomics) [[@lpalum](https://github.com/lpalum)] 1. [WePay](http://www.wepay.com) [[@criccomini](https://github.com/criccomini) & [@mtagle](https://github.com/mtagle)] 1. [WeTransfer](https://github.com/WeTransfer) [[@jochem](https://github.com/jochem)]
incubator-airflow git commit: [AIRFLOW-2798] Remove needless code from models.py
Repository: incubator-airflow
Updated Branches:
refs/heads/master f3b6b60c4 -> 53e021cde
[AIRFLOW-2798] Remove needless code from models.py
If `email_on_failure` or `email_on_failure` set to
TRUE and email available, there will be email sent
out on event of retry or failure.
In the implementation, there is a argument
`is_retry` passed to method `self.email_alert`.
However, inside this method, this argument is not
used at all. I believer the initial author of this
method was planning to differentiate the email to
be sent out, but for whatever reason this was not
implemented.
Given in the email to be sent out, there will be a
line "Try {try_number} out of {max_tries}", it
would be fine not to differentiate **retry email**
and **failure email**.
Closes #3640 from XD-DENG/patch-1
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/53e021cd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/53e021cd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/53e021cd
Branch: refs/heads/master
Commit: 53e021cde3502711d074988761a7de1eebbb5d6d
Parents: f3b6b60
Author: Xiaodong <11539188+xd-d...@users.noreply.github.com>
Authored: Wed Jul 25 19:09:38 2018 -0700
Committer: r39132
Committed: Wed Jul 25 19:09:41 2018 -0700
--
airflow/models.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/53e021cd/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 68a502d..d6a9199 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1755,7 +1755,7 @@ class TaskInstance(Base, LoggingMixin):
self.state = State.UP_FOR_RETRY
self.log.info('Marking task as UP_FOR_RETRY')
if task.email_on_retry and task.email:
-self.email_alert(error, is_retry=True)
+self.email_alert(error)
else:
self.state = State.FAILED
if task.retries:
@@ -1763,7 +1763,7 @@ class TaskInstance(Base, LoggingMixin):
else:
self.log.info('Marking task as FAILED.')
if task.email_on_failure and task.email:
-self.email_alert(error, is_retry=False)
+self.email_alert(error)
except Exception as e2:
self.log.error('Failed to send email to: %s', task.email)
self.log.exception(e2)
@@ -1927,7 +1927,7 @@ class TaskInstance(Base, LoggingMixin):
rendered_content = rt(attr, content, jinja_context)
setattr(task, attr, rendered_content)
-def email_alert(self, exception, is_retry=False):
+def email_alert(self, exception):
task = self.task
title = "Airflow alert: {self}".format(**locals())
exception = str(exception).replace('\n', '')
incubator-airflow git commit: [AIRFLOW-2758] Add a sensor for MongoDB
Repository: incubator-airflow
Updated Branches:
refs/heads/master cd75d4c85 -> af4d9614e
[AIRFLOW-2758] Add a sensor for MongoDB
This PR adds a sensor for MongoDB,
which waits for some document that
matches the given query to be
inserted to the specified collection.
Closes #3611 from sekikn/AIRFLOW-2758
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/af4d9614
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/af4d9614
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/af4d9614
Branch: refs/heads/master
Commit: af4d9614e7dd9bb437b05bc49baba472f8fdb4d0
Parents: cd75d4c
Author: Kengo Seki
Authored: Wed Jul 18 09:04:25 2018 -0700
Committer: r39132
Committed: Wed Jul 18 09:04:25 2018 -0700
--
.travis.yml| 1 +
airflow/contrib/sensors/mongo_sensor.py| 58 ++
tests/contrib/sensors/test_mongo_sensor.py | 65 +
3 files changed, 124 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/af4d9614/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 01c08d9..81e43fb 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -23,6 +23,7 @@ jdk:
- oraclejdk8
services:
- cassandra
+ - mongodb
- mysql
- postgresql
- rabbitmq
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/af4d9614/airflow/contrib/sensors/mongo_sensor.py
--
diff --git a/airflow/contrib/sensors/mongo_sensor.py
b/airflow/contrib/sensors/mongo_sensor.py
new file mode 100644
index 000..3ed49a6
--- /dev/null
+++ b/airflow/contrib/sensors/mongo_sensor.py
@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from airflow.contrib.hooks.mongo_hook import MongoHook
+from airflow.sensors.base_sensor_operator import BaseSensorOperator
+from airflow.utils.decorators import apply_defaults
+
+
+class MongoSensor(BaseSensorOperator):
+"""
+Checks for the existence of a document which
+matches the given query in MongoDB. Example:
+
+>>> mongo_sensor = MongoSensor(collection="coll",
+...query={"key": "value"},
+...mongo_conn_id="mongo_default",
+...task_id="mongo_sensor")
+"""
+template_fields = ('collection', 'query')
+
+@apply_defaults
+def __init__(self, collection, query, mongo_conn_id="mongo_default",
*args, **kwargs):
+"""
+Create a new MongoSensor
+
+:param collection: Target MongoDB collection.
+:type collection: string
+:param query: The query to find the target document.
+:type query: dict
+:param mongo_conn_id: The connection ID to use
+ when connecting to MongoDB.
+:type mongo_conn_id: string
+"""
+super(MongoSensor, self).__init__(*args, **kwargs)
+self.mongo_conn_id = mongo_conn_id
+self.collection = collection
+self.query = query
+
+def poke(self, context):
+self.log.info("Sensor check existence of the document "
+ "that matches the following query: %s", self.query)
+hook = MongoHook(self.mongo_conn_id)
+return hook.find(self.collection, self.query, find_one=True) is not
None
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/af4d9614/tests/contrib/sensors/test_mongo_sensor.py
--
diff --git a/tests/contrib/sensors/test_mongo_sensor.py
b/tests/contrib/sensors/test_mongo_sensor.py
new file mode 100644
index 000..876cb99
--- /dev/null
+++ b/tests/contrib/sensors/test_mongo_sensor.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+#
incubator-airflow git commit: [AIRFLOW-2705] Move class-level moto decorator to method-level
Repository: incubator-airflow
Updated Branches:
refs/heads/master d11593beb -> cd75d4c85
[AIRFLOW-2705] Move class-level moto decorator to method-level
Moto decorators at class-level make other
tests that sends HTTP request fail.
This PR moves them to method-level
so as to fix this problem.
Closes #3565 from sekikn/AIRFLOW-2705
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/cd75d4c8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/cd75d4c8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/cd75d4c8
Branch: refs/heads/master
Commit: cd75d4c85d9baa9fdb521a933082d95bb62974af
Parents: d11593b
Author: Kengo Seki
Authored: Wed Jul 18 08:54:19 2018 -0700
Committer: r39132
Committed: Wed Jul 18 08:54:19 2018 -0700
--
tests/contrib/hooks/test_redshift_hook.py| 15 ++-
.../sensors/test_aws_redshift_cluster_sensor.py | 9 -
2 files changed, 22 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cd75d4c8/tests/contrib/hooks/test_redshift_hook.py
--
diff --git a/tests/contrib/hooks/test_redshift_hook.py
b/tests/contrib/hooks/test_redshift_hook.py
index d5e791b..c69ed8a 100644
--- a/tests/contrib/hooks/test_redshift_hook.py
+++ b/tests/contrib/hooks/test_redshift_hook.py
@@ -31,10 +31,11 @@ except ImportError:
mock_redshift = None
-@mock_redshift
class TestRedshiftHook(unittest.TestCase):
def setUp(self):
configuration.load_test_config()
+
+def _create_clusters(self):
client = boto3.client('redshift', region_name='us-east-1')
client.create_cluster(
ClusterIdentifier='test_cluster',
@@ -52,7 +53,9 @@ class TestRedshiftHook(unittest.TestCase):
raise ValueError('AWS not properly mocked')
@unittest.skipIf(mock_redshift is None, 'mock_redshift package not
present')
+@mock_redshift
def
test_get_client_type_returns_a_boto3_client_of_the_requested_type(self):
+self._create_clusters()
hook = AwsHook(aws_conn_id='aws_default')
client_from_hook = hook.get_client_type('redshift')
@@ -60,7 +63,9 @@ class TestRedshiftHook(unittest.TestCase):
self.assertEqual(len(clusters), 2)
@unittest.skipIf(mock_redshift is None, 'mock_redshift package not
present')
+@mock_redshift
def
test_restore_from_cluster_snapshot_returns_dict_with_cluster_data(self):
+self._create_clusters()
hook = RedshiftHook(aws_conn_id='aws_default')
hook.create_cluster_snapshot('test_snapshot', 'test_cluster')
self.assertEqual(
@@ -70,27 +75,35 @@ class TestRedshiftHook(unittest.TestCase):
'test_cluster_3')
@unittest.skipIf(mock_redshift is None, 'mock_redshift package not
present')
+@mock_redshift
def test_delete_cluster_returns_a_dict_with_cluster_data(self):
+self._create_clusters()
hook = RedshiftHook(aws_conn_id='aws_default')
cluster = hook.delete_cluster('test_cluster_2')
self.assertNotEqual(cluster, None)
@unittest.skipIf(mock_redshift is None, 'mock_redshift package not
present')
+@mock_redshift
def test_create_cluster_snapshot_returns_snapshot_data(self):
+self._create_clusters()
hook = RedshiftHook(aws_conn_id='aws_default')
snapshot = hook.create_cluster_snapshot('test_snapshot_2',
'test_cluster')
self.assertNotEqual(snapshot, None)
@unittest.skipIf(mock_redshift is None, 'mock_redshift package not
present')
+@mock_redshift
def test_cluster_status_returns_cluster_not_found(self):
+self._create_clusters()
hook = RedshiftHook(aws_conn_id='aws_default')
status = hook.cluster_status('test_cluster_not_here')
self.assertEqual(status, 'cluster_not_found')
@unittest.skipIf(mock_redshift is None, 'mock_redshift package not
present')
+@mock_redshift
def test_cluster_status_returns_available_cluster(self):
+self._create_clusters()
hook = RedshiftHook(aws_conn_id='aws_default')
status = hook.cluster_status('test_cluster')
self.assertEqual(status, 'available')
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/cd75d4c8/tests/contrib/sensors/test_aws_redshift_cluster_sensor.py
--
diff --git a/tests/contrib/sensors/test_aws_redshift_cluster_sensor.py
b/tests/contrib/sensors/test_aws_redshift_cluster_sensor.py
index 5bee24a..921b2da 100644
--- a/tests/contrib/sensors/test_aws_redshift_cluster_sensor.py
+++ b/tests/contrib/sensors/test_aws_redshift_cluster_sensor.py
@@ -31
incubator-airflow git commit: [AIRFLOW-XXX] Add Publicis Pixelpark to README
Repository: incubator-airflow Updated Branches: refs/heads/master 2c20ae81f -> d11593beb [AIRFLOW-XXX] Add Publicis Pixelpark to README Closes #3556 from feluelle/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d11593be Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d11593be Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d11593be Branch: refs/heads/master Commit: d11593bebf9e5551bd8bd3941b528faf57cc19f4 Parents: 2c20ae8 Author: Felix Authored: Tue Jul 17 15:49:17 2018 -0700 Committer: r39132 Committed: Tue Jul 17 15:49:22 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d11593be/README.md -- diff --git a/README.md b/README.md index 184bf9f..e1de8ff 100644 --- a/README.md +++ b/README.md @@ -208,6 +208,7 @@ Currently **officially** using Airflow: 1. [PMC](https://pmc.com/) [[@andrewm4894](https://github.com/andrewm4894)] 1. [Postmates](http://www.postmates.com) [[@syeoryn](https://github.com/syeoryn)] 1. [Pronto Tools](http://www.prontotools.io/) [[@zkan](https://github.com/zkan) & [@mesodiar](https://github.com/mesodiar)] +1. [Publicis Pixelpark](https://www.publicispixelpark.de/) [[@feluelle](https://github.com/feluelle)] 1. [PubNub](https://pubnub.com) [[@jzucker2](https://github.com/jzucker2)] 1. [Qplum](https://qplum.co) [[@manti](https://github.com/manti)] 1. [Quantopian](https://www.quantopian.com/) [[@eronarn](http://github.com/eronarn)]
incubator-airflow git commit: [AIRFLOW-2646] Fix setup.py not to install snakebite on Python3
Repository: incubator-airflow Updated Branches: refs/heads/master ecf68c7d3 -> 105ac1505 [AIRFLOW-2646] Fix setup.py not to install snakebite on Python3 setup.py has a logic to avoid installing snakebite on Python3, but it doesn't work. This is because the variable devel_all is now a tuple of lists. This PR fixes that variable to be a flat list and makes the logic work as expected. Closes #3522 from sekikn/AIRFLOW-2646 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/105ac150 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/105ac150 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/105ac150 Branch: refs/heads/master Commit: 105ac150580af0a1a64bb5e0417fc57b2a0e49bb Parents: ecf68c7 Author: Kengo Seki Authored: Tue Jun 19 17:09:00 2018 -0700 Committer: r39132 Committed: Tue Jun 19 17:09:00 2018 -0700 -- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/105ac150/setup.py -- diff --git a/setup.py b/setup.py index 0b012ee..2f22ee8 100644 --- a/setup.py +++ b/setup.py @@ -223,7 +223,8 @@ devel_hadoop = devel_minreq + hive + hdfs + webhdfs + kerberos devel_all = (sendgrid + devel + all_dbs + doc + samba + s3 + slack + crypto + oracle + docker + ssh + kubernetes + celery + azure_blob_storage + redis + gcp_api + datadog + zendesk + jdbc + ldap + kerberos + password + webhdfs + jenkins + - druid + pinot + segment + snowflake + elasticsearch + azure_data_lake, atlas) + druid + pinot + segment + snowflake + elasticsearch + azure_data_lake + + atlas) # Snakebite & Google Cloud Dataflow are not Python 3 compatible :'( if PY3:
incubator-airflow git commit: [AIRFLOW-XXX] Adding REA Group to readme
Repository: incubator-airflow Updated Branches: refs/heads/master 528859b81 -> 7bdf6162f [AIRFLOW-XXX] Adding REA Group to readme Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7bdf6162 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7bdf6162 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7bdf6162 Branch: refs/heads/master Commit: 7bdf6162fb1f86fcbd970d242800f1d4ae2f1d45 Parents: 528859b Author: r39132 Authored: Mon Jun 18 15:44:12 2018 -0700 Committer: r39132 Committed: Mon Jun 18 15:47:08 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7bdf6162/README.md -- diff --git a/README.md b/README.md index 3726c54..e68d26c 100644 --- a/README.md +++ b/README.md @@ -211,6 +211,7 @@ Currently **officially** using Airflow: 1. [Qubole](https://qubole.com) [[@msumit](https://github.com/msumit)] 1. [Quizlet](https://quizlet.com) [[@quizlet](https://github.com/quizlet)] 1. [Quora](https://www.quora.com/) +1. [REA Group](https://www.rea-group.com/) 1. [Reddit](https://www.reddit.com/) [[@reddit](https://github.com/reddit/)] 1. [Robinhood](https://robinhood.com) [[@vineet-rh](https://github.com/vineet-rh)] 1. [Scaleway](https://scaleway.com) [[@kdeldycke](https://github.com/kdeldycke)]
incubator-airflow git commit: closes apache/incubator-airflow#3507 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master 14a77a701 -> 528859b81 closes apache/incubator-airflow#3507 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/528859b8 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/528859b8 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/528859b8 Branch: refs/heads/master Commit: 528859b817ccdd2c541458cd70f865338fb36c9b Parents: 14a77a7 Author: r39132 Authored: Mon Jun 18 15:28:50 2018 -0700 Committer: r39132 Committed: Mon Jun 18 15:28:50 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1021] Fix double login for new users with LDAP
Repository: incubator-airflow
Updated Branches:
refs/heads/master 016c0665d -> 9c67ee842
[AIRFLOW-1021] Fix double login for new users with LDAP
By adding the new user to the session and commit
before merging, the correct user is returned by
`load_user(userid)` instead of `None` due to no
`id` assoicated with that user.
Closes #2778 from wolfier/AIRFLOW-1021
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9c67ee84
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9c67ee84
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9c67ee84
Branch: refs/heads/master
Commit: 9c67ee84296b649ccb998e5b85187b956df9a757
Parents: 016c066
Author: Alan Ma
Authored: Thu Jun 7 14:58:38 2018 -0700
Committer: r39132
Committed: Thu Jun 7 14:58:43 2018 -0700
--
airflow/contrib/auth/backends/ldap_auth.py | 3 ++-
tests/core.py | 3 +++
2 files changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9c67ee84/airflow/contrib/auth/backends/ldap_auth.py
--
diff --git a/airflow/contrib/auth/backends/ldap_auth.py
b/airflow/contrib/auth/backends/ldap_auth.py
index 177a6de..eefaa12 100644
--- a/airflow/contrib/auth/backends/ldap_auth.py
+++ b/airflow/contrib/auth/backends/ldap_auth.py
@@ -303,9 +303,10 @@ def login(self, request, session=None):
user = models.User(
username=username,
is_superuser=False)
+session.add(user)
-session.merge(user)
session.commit()
+session.merge(user)
flask_login.login_user(LdapUser(user))
session.commit()
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9c67ee84/tests/core.py
--
diff --git a/tests/core.py b/tests/core.py
index 0312fed..c3adb3c 100644
--- a/tests/core.py
+++ b/tests/core.py
@@ -2042,6 +2042,9 @@ class WebLdapAuthTest(unittest.TestCase):
response = self.login('dataprofiler', 'dataprofiler')
self.assertIn('Data Profiling', response.data.decode('utf-8'))
+response = self.logout()
+self.assertIn('form-signin', response.data.decode('utf-8'))
+
response = self.login('superuser', 'superuser')
self.assertIn('Connections', response.data.decode('utf-8'))
incubator-airflow git commit: [AIRFLOW-2561] Fix typo in EmailOperator
Repository: incubator-airflow
Updated Branches:
refs/heads/master b8487cd44 -> 3358551c8
[AIRFLOW-2561] Fix typo in EmailOperator
There's a typo in the params in send_email.
It should be mime_charset instead of mine_charset.
Closes #3468 from wolfier/AIRFLOW-2561
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3358551c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3358551c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3358551c
Branch: refs/heads/master
Commit: 3358551c8e73d9019900f7a85f18ebfd88591450
Parents: b8487cd
Author: Alan Ma
Authored: Wed Jun 6 16:25:21 2018 -0700
Committer: r39132
Committed: Wed Jun 6 16:25:21 2018 -0700
--
airflow/operators/email_operator.py| 2 +-
tests/operators/test_email_operator.py | 65 +
2 files changed, 66 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3358551c/airflow/operators/email_operator.py
--
diff --git a/airflow/operators/email_operator.py
b/airflow/operators/email_operator.py
index 4f5396f..ae176ca 100644
--- a/airflow/operators/email_operator.py
+++ b/airflow/operators/email_operator.py
@@ -75,4 +75,4 @@ class EmailOperator(BaseOperator):
def execute(self, context):
send_email(self.to, self.subject, self.html_content,
files=self.files, cc=self.cc, bcc=self.bcc,
- mime_subtype=self.mime_subtype,
mine_charset=self.mime_charset)
+ mime_subtype=self.mime_subtype,
mime_charset=self.mime_charset)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3358551c/tests/operators/test_email_operator.py
--
diff --git a/tests/operators/test_email_operator.py
b/tests/operators/test_email_operator.py
new file mode 100644
index 000..a19f450
--- /dev/null
+++ b/tests/operators/test_email_operator.py
@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from __future__ import print_function, unicode_literals
+
+import datetime
+import mock
+import unittest
+
+from airflow import configuration, DAG
+from airflow.operators.email_operator import EmailOperator
+from airflow.utils import timezone
+
+DEFAULT_DATE = timezone.datetime(2016, 1, 1)
+END_DATE = timezone.datetime(2016, 1, 2)
+INTERVAL = datetime.timedelta(hours=12)
+FROZEN_NOW = timezone.datetime(2016, 1, 2, 12, 1, 1)
+
+send_email_test = mock.Mock()
+
+
+class TestEmailOperator(unittest.TestCase):
+
+def setUp(self):
+super(TestEmailOperator, self).setUp()
+configuration.load_test_config()
+self.dag = DAG(
+'test_dag',
+default_args={
+'owner': 'airflow',
+'start_date': DEFAULT_DATE},
+schedule_interval=INTERVAL)
+self.addCleanup(self.dag.clear)
+
+def _run_as_operator(self, **kwargs):
+task = EmailOperator(
+to='airf...@example.com',
+subject='Test Run',
+html_content='The quick brown fox jumps over the lazy dog',
+task_id='task',
+dag=self.dag,
+**kwargs)
+task.run(start_date=DEFAULT_DATE, end_date=DEFAULT_DATE)
+
+def test_execute(self):
+configuration.conf.set('email', 'EMAIL_BACKEND',
+
'tests.operators.test_email_operator.send_email_test')
+self._run_as_operator()
+send_email_test.assert_called_once()
incubator-airflow git commit: [AIRFLOW-XXX] Fix doc typos
Repository: incubator-airflow
Updated Branches:
refs/heads/master b220fe60d -> 817296a7b
[AIRFLOW-XXX] Fix doc typos
Closes #3459 from ccayg-sainsburys/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/817296a7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/817296a7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/817296a7
Branch: refs/heads/master
Commit: 817296a7be58a1ef776e77f67b5947d495ad4e1d
Parents: b220fe6
Author: Charles Caygill
Authored: Mon Jun 4 11:15:35 2018 -0700
Committer: r39132
Committed: Mon Jun 4 11:15:38 2018 -0700
--
docs/faq.rst | 2 +-
docs/plugins.rst | 2 +-
docs/profiling.rst | 2 +-
docs/security.rst | 6 +++---
4 files changed, 6 insertions(+), 6 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/817296a7/docs/faq.rst
--
diff --git a/docs/faq.rst b/docs/faq.rst
index 33b4d5a..275c26b 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -46,7 +46,7 @@ Here are some of the common causes:
running. You can bulk view the list of DagRuns and alter states by clicking
on the schedule tag for a DAG.
-- Is the ``concurrency`` parameter of your DAG reached? ``concurency`` defines
+- Is the ``concurrency`` parameter of your DAG reached? ``concurrency`` defines
how many ``running`` task instances a DAG is allowed to have, beyond which
point things get queued.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/817296a7/docs/plugins.rst
--
diff --git a/docs/plugins.rst b/docs/plugins.rst
index eaba1a1..a21f100 100644
--- a/docs/plugins.rst
+++ b/docs/plugins.rst
@@ -127,7 +127,7 @@ definitions in Airflow.
return self.render("test_plugin/test.html", content="Hello
galaxy!")
v = TestView(category="Test Plugin", name="Test View")
-# Creating a flask blueprint to intergrate the templates and static folder
+# Creating a flask blueprint to integrate the templates and static folder
bp = Blueprint(
"test_plugin", __name__,
template_folder='templates', # registers airflow/plugins/templates as
a Jinja template folder
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/817296a7/docs/profiling.rst
--
diff --git a/docs/profiling.rst b/docs/profiling.rst
index 93e6b6b..0910233 100644
--- a/docs/profiling.rst
+++ b/docs/profiling.rst
@@ -17,7 +17,7 @@ Charts
--
A simple UI built on top of flask-admin and highcharts allows building
data visualizations and charts easily. Fill in a form with a label, SQL,
-chart type, pick a source database from your environment's connectons,
+chart type, pick a source database from your environment's connections,
select a few other options, and save it for later use.
You can even use the same templating and macros available when writing
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/817296a7/docs/security.rst
--
diff --git a/docs/security.rst b/docs/security.rst
index 6b3f933..5d459de 100644
--- a/docs/security.rst
+++ b/docs/security.rst
@@ -27,7 +27,7 @@ uses bcrypt before storing passwords.
auth_backend = airflow.contrib.auth.backends.password_auth
When password auth is enabled, an initial user credential will need to be
created before anyone can login. An initial
-user was not created in the migrations for this authenication backend to
prevent default Airflow installations from
+user was not created in the migrations for this authentication backend to
prevent default Airflow installations from
attack. Creating a new user has to be done via a Python REPL on the same
machine Airflow is installed.
.. code-block:: bash
@@ -263,7 +263,7 @@ backend. In order to setup an application:
2. Select 'Applications' from the left hand nav
3. Select the 'Developer Applications' tab
4. Click 'Register new application'
-5. Fill in the required information (the 'Authorization callback URL' must be
fully qualifed e.g. http://airflow.example.com/example/ghe_oauth/callback)
+5. Fill in the required information (the 'Authorization callback URL' must be
fully qualified e.g. http://airflow.example.com/example/ghe_oauth/callback)
6. Click 'Register application'
7. Copy 'Client ID', 'Client Secret', and your callback route to your
airflow.cfg according to the above example
@@ -296,7 +296,7 @@ backend. In order to setup an application:
2. Select 'Credentials' from the left hand nav
3. Click 'Create credentials' and choose 'OAuth client ID'
4. Choose 'Web application'
-5. Fil
incubator-airflow git commit: [AIRFLOW-2553] Add webserver.pid to .gitignore
Repository: incubator-airflow Updated Branches: refs/heads/master ef7fe870e -> 2ed618883 [AIRFLOW-2553] Add webserver.pid to .gitignore Closes #3451 from r39132/update_gitignore Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2ed61888 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2ed61888 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2ed61888 Branch: refs/heads/master Commit: 2ed6188838d78ec8dbbb1bdf65c623ee5c6426dc Parents: ef7fe87 Author: r39132 Authored: Fri Jun 1 19:19:44 2018 -0700 Committer: r39132 Committed: Fri Jun 1 19:19:44 2018 -0700 -- .gitignore | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2ed61888/.gitignore -- diff --git a/.gitignore b/.gitignore index a42962f..0e53aad 100644 --- a/.gitignore +++ b/.gitignore @@ -14,6 +14,7 @@ unittests.db airflow/git_version airflow/www/static/coverage/ logs/ +airflow-webserver.pid # Byte-compiled / optimized / DLL files __pycache__/
incubator-airflow git commit: [AIRFLOW-1863][AIRFLOW-2529] Add dag run selection widgets to gantt view
Repository: incubator-airflow
Updated Branches:
refs/heads/master 2fb84a89b -> ef7fe870e
[AIRFLOW-1863][AIRFLOW-2529] Add dag run selection widgets to gantt view
Add same widgets to filter and select dag run
known from graph view
to the gantt chart view. Extract common code to
handle request
parameters and DB query.
Closes #3450 from seelmann/AIRFLOW-1863-gantt-view
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ef7fe870
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ef7fe870
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ef7fe870
Branch: refs/heads/master
Commit: ef7fe870e0a77398f1579eeae027da6cda083704
Parents: 2fb84a8
Author: Stefan Seelmann
Authored: Fri Jun 1 18:59:22 2018 -0700
Committer: r39132
Committed: Fri Jun 1 18:59:22 2018 -0700
--
airflow/www/forms.py | 6 +-
airflow/www/templates/airflow/gantt.html | 2 +
airflow/www/views.py | 120 +++---
airflow/www_rbac/forms.py | 7 +-
airflow/www_rbac/templates/airflow/gantt.html | 2 +
airflow/www_rbac/views.py | 119 +++---
tests/www/test_views.py | 151 +-
tests/www_rbac/test_views.py | 177 ++---
8 files changed, 377 insertions(+), 207 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ef7fe870/airflow/www/forms.py
--
diff --git a/airflow/www/forms.py b/airflow/www/forms.py
index d6fbe13..42bf470 100644
--- a/airflow/www/forms.py
+++ b/airflow/www/forms.py
@@ -29,7 +29,7 @@ from flask_wtf import FlaskForm
class DateTimeForm(FlaskForm):
-# Date filter form needed for gantt and graph view
+# Date filter form for task views
execution_date = DateTimeField(
"Execution date", widget=DateTimePickerWidget())
@@ -47,3 +47,7 @@ class DateTimeWithNumRunsForm(FlaskForm):
(365, "365"),
))
+
+class DateTimeWithNumRunsWithDagRunsForm(DateTimeWithNumRunsForm):
+# Date time and number of runs and dag runs form for graph and gantt view
+execution_date = SelectField("DAG run")
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ef7fe870/airflow/www/templates/airflow/gantt.html
--
diff --git a/airflow/www/templates/airflow/gantt.html
b/airflow/www/templates/airflow/gantt.html
index ecf9b6d..01d3cd8 100644
--- a/airflow/www/templates/airflow/gantt.html
+++ b/airflow/www/templates/airflow/gantt.html
@@ -28,6 +28,8 @@
{{ super() }}
+Base date: {{ form.base_date(class_="form-control") }}
+Number of runs: {{ form.num_runs(class_="form-control") }}
Run:
{{ form.execution_date(class_="form-control") | safe }}
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ef7fe870/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 3d09670..a16b685 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -89,7 +89,8 @@ from airflow.utils.dates import infer_time_unit,
scale_time_units, parse_executi
from airflow.utils.timezone import datetime
from airflow.utils.net import get_hostname
from airflow.www import utils as wwwutils
-from airflow.www.forms import DateTimeForm, DateTimeWithNumRunsForm
+from airflow.www.forms import (DateTimeForm, DateTimeWithNumRunsForm,
+ DateTimeWithNumRunsWithDagRunsForm)
from airflow.www.validators import GreaterEqualThan
QUERY_LIMIT = 10
@@ -318,6 +319,57 @@ def get_chart_height(dag):
return 600 + len(dag.tasks) * 10
+def get_date_time_num_runs_dag_runs_form_data(request, session, dag):
+dttm = request.args.get('execution_date')
+if dttm:
+dttm = pendulum.parse(dttm)
+else:
+dttm = dag.latest_execution_date or timezone.utcnow()
+
+base_date = request.args.get('base_date')
+if base_date:
+base_date = timezone.parse(base_date)
+else:
+# The DateTimeField widget truncates milliseconds and would loose
+# the first dag run. Round to next second.
+base_date = (dttm + timedelta(seconds=1)).replace(microsecond=0)
+
+default_dag_run = conf.getint('webserver',
'default_dag_run_display_number')
+num_runs = request.args.get('num_runs')
+num_runs = int(num_runs) if num_runs else default_dag_run
+
+DR = models.DagRun
+drs = (
+session.query(DR)
+.filter(
+DR.dag_id == dag.dag_id,
+DR.execution_date <= base_date)
+.order_by(desc(DR.execution_
incubator-airflow git commit: closes apache/incubator-airflow#3328 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master 86d096004 -> 2fb84a89b closes apache/incubator-airflow#3328 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2fb84a89 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2fb84a89 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2fb84a89 Branch: refs/heads/master Commit: 2fb84a89b69a2007f546477f67e8d824232b438e Parents: 86d0960 Author: r39132 Authored: Fri Jun 1 15:37:56 2018 -0700 Committer: r39132 Committed: Fri Jun 1 15:37:56 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2544][AIRFLOW-1967] Guard against next major release of Celery, Flower
Repository: incubator-airflow Updated Branches: refs/heads/master a41b75eba -> 8a5e5132d [AIRFLOW-2544][AIRFLOW-1967] Guard against next major release of Celery, Flower Closes #3439 from tedmiston/celery-flower-guard Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8a5e5132 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8a5e5132 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8a5e5132 Branch: refs/heads/master Commit: 8a5e5132d45f98f3a36f9cb7f582867c732d62bf Parents: a41b75e Author: Taylor D. Edmiston Authored: Fri Jun 1 09:49:52 2018 -0700 Committer: r39132 Committed: Fri Jun 1 09:49:52 2018 -0700 -- setup.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8a5e5132/setup.py -- diff --git a/setup.py b/setup.py index 83d1752..2a11f74 100644 --- a/setup.py +++ b/setup.py @@ -116,8 +116,8 @@ azure_data_lake = [ ] cassandra = ['cassandra-driver>=3.13.0'] celery = [ -'celery>=4.1.1', -'flower>=0.7.3' +'celery>=4.1.1, <4.2.0', +'flower>=0.7.3, <1.0' ] cgroups = [ 'cgroupspy>=0.1.4',
incubator-airflow git commit: [AIRFLOW-XXX] Add Yieldr to who is using airflow
Repository: incubator-airflow Updated Branches: refs/heads/master 7db181836 -> a41b75eba [AIRFLOW-XXX] Add Yieldr to who is using airflow Closes #3448 from ggeorgiadis/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a41b75eb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a41b75eb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a41b75eb Branch: refs/heads/master Commit: a41b75eba9739645cde5778fd6ca1a4e5ea01e2e Parents: 7db1818 Author: Georgios Authored: Fri Jun 1 09:46:45 2018 -0700 Committer: r39132 Committed: Fri Jun 1 09:46:49 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a41b75eb/README.md -- diff --git a/README.md b/README.md index 0f94de0..3726c54 100644 --- a/README.md +++ b/README.md @@ -247,6 +247,7 @@ Currently **officially** using Airflow: 1. [Xero](https://www.xero.com/) [[@yan9yu](https://github.com/yan9yu)] 1. [Xoom](https://www.xoom.com/) 1. [Yahoo!](https://www.yahoo.com/) +1. [Yieldr](https://www.yieldr.com/) [[@ggeorgiadis](https://github.com/ggeorgiadis)] 1. [Zapier](https://www.zapier.com) [[@drknexus](https://github.com/drknexus) & [@statwonk](https://github.com/statwonk)] 1. [Zego](https://www.zego.com/) [[@ruimffl](https://github.com/ruimffl)] 1. [Zendesk](https://www.github.com/zendesk)
incubator-airflow git commit: [AIRFLOW-2547] Describe how to run tests using Docker
Repository: incubator-airflow Updated Branches: refs/heads/master b81bd08a3 -> 7db181836 [AIRFLOW-2547] Describe how to run tests using Docker For Airflow it can be a bit tricky to test the code. Add a description on how to build a clean enviroment from scratch Closes #3445 from Fokko/fd-local-devel-environment Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7db18183 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7db18183 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7db18183 Branch: refs/heads/master Commit: 7db18183677d96e37f4671aa846815d3ecc7e58c Parents: b81bd08 Author: Fokko Driesprong Authored: Fri Jun 1 09:44:04 2018 -0700 Committer: r39132 Committed: Fri Jun 1 09:44:04 2018 -0700 -- CONTRIBUTING.md | 39 ++- setup.py| 4 ++-- 2 files changed, 40 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7db18183/CONTRIBUTING.md -- diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fd9958b..e45060b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -79,7 +79,44 @@ extras to build the full API reference. ## Development and Testing -### Setting up a development environment +### Set up a development env using Docker + +Go to your Airflow directory and start a new docker container. You can choose between Python 2 or 3, whatever you prefer. + +``` +# Start docker in your Airflow directory +docker run -t -i -v `pwd`:/airflow/ python:2 bash + +# Go to the Airflow directory +cd /airflow/ + +# Install Airflow with all the required dependencies, +# including the devel which will provide the development tools +pip install -e ".[hdfs,hive,druid,devel]" + +# Init the database +airflow initdb + +nosetests -v tests/hooks/test_druid_hook.py + + test_get_first_record (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok + test_get_records (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok + test_get_uri (tests.hooks.test_druid_hook.TestDruidDbApiHook) ... ok + test_get_conn_url (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_gone_wrong (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_ok (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_timeout (tests.hooks.test_druid_hook.TestDruidHook) ... ok + test_submit_unknown_response (tests.hooks.test_druid_hook.TestDruidHook) ... ok + + -- + Ran 8 tests in 3.036s + + OK +``` + +The Airflow code is mounted inside of the Docker container, so if you change something using your favorite IDE, you can directly test is in the container. + +### Set up a development env using Virtualenv Please install python(2.7.x or 3.4.x), mysql, and libxml by using system-level package managers like yum, apt-get for Linux, or homebrew for Mac OS at first. http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7db18183/setup.py -- diff --git a/setup.py b/setup.py index 273d1ce..83d1752 100644 --- a/setup.py +++ b/setup.py @@ -157,7 +157,6 @@ hive = [ 'pyhive>=0.1.3', 'impyla>=0.13.3', 'thrift_sasl==0.2.1', -'unicodecsv>=0.14.1' ] jdbc = ['jaydebeapi>=1.1.1'] jenkins = ['python-jenkins>=0.4.15'] @@ -169,7 +168,7 @@ kerberos = ['pykerberos>=1.1.13', kubernetes = ['kubernetes>=3.0.0', 'cryptography>=2.0.0'] ldap = ['ldap3>=0.9.9.1'] -mssql = ['pymssql>=2.1.1', 'unicodecsv>=0.14.1'] +mssql = ['pymssql>=2.1.1'] mysql = ['mysqlclient>=1.3.6'] oracle = ['cx_Oracle>=5.1.2'] password = [ @@ -279,6 +278,7 @@ def do_setup(): 'tenacity==4.8.0', 'thrift>=0.9.2', 'tzlocal>=1.4', +'unicodecsv>=0.14.1', 'werkzeug>=0.14.1, <0.15.0', 'zope.deprecation>=4.0, <5.0', ],
incubator-airflow git commit: closes apache/incubator-airflow#3310 *Fixed in another PR.*
Repository: incubator-airflow Updated Branches: refs/heads/master 3ed25a945 -> c6681681d closes apache/incubator-airflow#3310 *Fixed in another PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c6681681 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c6681681 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c6681681 Branch: refs/heads/master Commit: c6681681d41e9b914b857587823190fb506274c7 Parents: 3ed25a9 Author: r39132 Authored: Thu May 31 14:33:28 2018 -0700 Committer: r39132 Committed: Thu May 31 14:33:28 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#3403 *Obsolete PR.*
Repository: incubator-airflow Updated Branches: refs/heads/master 3bdb34e77 -> cedcdb1f8 closes apache/incubator-airflow#3403 *Obsolete PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/cedcdb1f Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/cedcdb1f Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/cedcdb1f Branch: refs/heads/master Commit: cedcdb1f86e04f03cef13a727bdca00bdf7243be Parents: 3bdb34e Author: r39132 Authored: Wed May 23 14:02:37 2018 -0700 Committer: r39132 Committed: Wed May 23 14:02:37 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2472] Implement MySqlHook.bulk_dump
Repository: incubator-airflow
Updated Branches:
refs/heads/master f1ac67bdc -> 3bdb34e77
[AIRFLOW-2472] Implement MySqlHook.bulk_dump
Implement MySqlHook.bulk_dump since the opposite
operation bulk_load is already implemented.
This PR also addresses some flake8 warnings.
Closes #3385 from sekikn/AIRFLOW-2472
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3bdb34e7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3bdb34e7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3bdb34e7
Branch: refs/heads/master
Commit: 3bdb34e7775fdb85e3dd6cd92a47f91c80b1f67b
Parents: f1ac67b
Author: Kengo Seki
Authored: Wed May 23 13:58:04 2018 -0700
Committer: r39132
Committed: Wed May 23 13:58:04 2018 -0700
--
airflow/hooks/mysql_hook.py | 12
airflow/utils/tests.py | 8 +++
tests/hooks/test_hive_hook.py | 9 +--
tests/operators/operators.py| 75 ++--
tests/operators/test_redshift_to_s3_operator.py | 11 +--
tests/operators/test_s3_to_redshift_operator.py | 8 +--
6 files changed, 79 insertions(+), 44 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3bdb34e7/airflow/hooks/mysql_hook.py
--
diff --git a/airflow/hooks/mysql_hook.py b/airflow/hooks/mysql_hook.py
index d202e55..f52b60c 100644
--- a/airflow/hooks/mysql_hook.py
+++ b/airflow/hooks/mysql_hook.py
@@ -89,6 +89,18 @@ class MySqlHook(DbApiHook):
""".format(**locals()))
conn.commit()
+def bulk_dump(self, table, tmp_file):
+"""
+Dumps a database table into a tab-delimited file
+"""
+conn = self.get_conn()
+cur = conn.cursor()
+cur.execute("""
+SELECT * INTO OUTFILE '{tmp_file}'
+FROM {table}
+""".format(**locals()))
+conn.commit()
+
@staticmethod
def _serialize_cell(cell, conn):
"""
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3bdb34e7/airflow/utils/tests.py
--
diff --git a/airflow/utils/tests.py b/airflow/utils/tests.py
index 9982a07..6f29ffc 100644
--- a/airflow/utils/tests.py
+++ b/airflow/utils/tests.py
@@ -17,8 +17,10 @@
# specific language governing permissions and limitations
# under the License.
+import re
import unittest
+
def skipUnlessImported(module, obj):
import importlib
try:
@@ -29,3 +31,9 @@ def skipUnlessImported(module, obj):
obj in dir(m),
"Skipping test because {} could not be imported from {}".format(
obj, module))
+
+
+def assertEqualIgnoreMultipleSpaces(case, first, second, msg=None):
+def _trim(s):
+re.sub("\s+", " ", s.strip())
+return case.assertEqual(_trim(first), _trim(second), msg)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3bdb34e7/tests/hooks/test_hive_hook.py
--
diff --git a/tests/hooks/test_hive_hook.py b/tests/hooks/test_hive_hook.py
index d090086..690b5ad 100644
--- a/tests/hooks/test_hive_hook.py
+++ b/tests/hooks/test_hive_hook.py
@@ -22,7 +22,6 @@ import datetime
import itertools
import pandas as pd
import random
-import re
import mock
import unittest
@@ -34,6 +33,7 @@ from airflow.exceptions import AirflowException
from airflow.hooks.hive_hooks import HiveCliHook, HiveMetastoreHook
from airflow import DAG, configuration, operators
from airflow.utils import timezone
+from airflow.utils.tests import assertEqualIgnoreMultipleSpaces
configuration.load_test_config()
@@ -188,12 +188,7 @@ class TestHiveCliHook(unittest.TestCase):
STORED AS textfile
;
"""
-
-def _trim(s):
-return re.sub("\s+", " ", s.strip())
-
-self.assertEqual(_trim(mock_run_cli.call_args_list[0][0][0]),
- _trim(query))
+assertEqualIgnoreMultipleSpaces(self,
mock_run_cli.call_args_list[0][0][0], query)
class TestHiveMetastoreHook(HiveEnvironmentTest):
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3bdb34e7/tests/operators/operators.py
--
diff --git a/tests/operators/operators.py b/tests/operators/operators.py
index 9fc2c93..f795fd0 100644
--- a/tests/operators/operators.py
+++ b/tests/operators/operators.py
@@ -23,10 +23,12 @@ from airflow import DAG, configuration, operators
from airflow.utils.tests import skipUnlessImported
from airflow.utils import timezone
-configuration.load_test_config()
-
+import os
+import mock
import
incubator-airflow git commit: Deleting specific names
Repository: incubator-airflow Updated Branches: refs/heads/master 4ce250295 -> f8e9c688e Deleting specific names Deleting specific names because I no longer work at Xoom. Closes #3372 from gepser/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f8e9c688 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f8e9c688 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f8e9c688 Branch: refs/heads/master Commit: f8e9c688e7b4f39a3209c08b535fcc1061ece89a Parents: 4ce2502 Author: Gepser Hoil Authored: Thu May 17 08:07:20 2018 -0700 Committer: r39132 Committed: Thu May 17 08:07:20 2018 -0700 -- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f8e9c688/README.md -- diff --git a/README.md b/README.md index ac70f92..08a15d6 100644 --- a/README.md +++ b/README.md @@ -242,7 +242,7 @@ Currently **officially** using Airflow: 1. [WiseBanyan](https://wisebanyan.com/) 1. [Wooga](https://www.wooga.com/) 1. [Xero](https://www.xero.com/) [[@yan9yu](https://github.com/yan9yu)] -1. [Xoom](https://www.xoom.com/india/send-money) [[@gepser](https://github.com/gepser) & [@omarvides](https://github.com/omarvides)] +1. [Xoom](https://www.xoom.com/) 1. [Yahoo!](https://www.yahoo.com/) 1. [Zapier](https://www.zapier.com) [[@drknexus](https://github.com/drknexus) & [@statwonk](https://github.com/statwonk)] 1. [Zego](https://www.zego.com/) [[@ruimffl](https://github.com/ruimffl)]
incubator-airflow git commit: [AIRFLOW-XXX] Fix order of companies
Repository: incubator-airflow
Updated Branches:
refs/heads/master 648b14b4d -> ee81a6ad8
[AIRFLOW-XXX] Fix order of companies
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
- https://issues.apache.org/jira/browse/AIRFLOW-
XXX
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Fix the alphabetic order of companies in the
README
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason: N/A -- documentation only
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3363 from r39132/fix_order_of_company_list
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ee81a6ad
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ee81a6ad
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ee81a6ad
Branch: refs/heads/master
Commit: ee81a6ad8bcc9a7aa6bb3ddbf88c86a1d64ceceb
Parents: 648b14b
Author: r39132
Authored: Tue May 15 18:51:22 2018 -0700
Committer: r39132
Committed: Tue May 15 18:51:22 2018 -0700
--
README.md | 16
1 file changed, 8 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ee81a6ad/README.md
--
diff --git a/README.md b/README.md
index 743a195..ac70f92 100644
--- a/README.md
+++ b/README.md
@@ -83,9 +83,9 @@ Committers:
Currently **officially** using Airflow:
-1. [Airbnb](http://airbnb.io/)
[[@mistercrunch](https://github.com/mistercrunch),
[@artwr](https://github.com/artwr)]
1. [AdBOOST](https://www.adboost.sk) [[AdBOOST](https://github.com/AdBOOST)]
1. [Agari](https://github.com/agaridata) [[@r39132](https://github.com/r39132)]
+1. [Airbnb](http://airbnb.io/)
[[@mistercrunch](https://github.com/mistercrunch),
[@artwr](https://github.com/artwr)]
1. [Alan](https://alan.eu) [[@charles-go](https://github.com/charles-go)]
1. [allegro.pl](http://allegro.tech/) [[@kretes](https://github.com/kretes)]
1. [AltX](https://www.getaltx.com/about)
[[@pedromduarte](https://github.com/pedromduarte)]
@@ -94,17 +94,17 @@ Currently **officially** using Airflow:
1. [Astronomer](http://www.astronomer.io)
[[@schnie](https://github.com/schnie),
[@andscoop](https://github.com/andscoop),
[@tedmiston](https://github.com/tedmiston),
[@benjamingregory](https://github.com/benjamingregory)]
1. [Auth0](https://auth0.com) [[@sicarul](https://github.com/sicarul)]
1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)]
+1. [Azri Solutions](http://www.azrisolutions.com/)
[[@userimack](https://github.com/userimack)]
1. [BalanceHero](http://truebalance.io/)
[[@swalloow](https://github.com/swalloow)]
1. [Banco de Formaturas](https://www.bancodeformaturas.com.br)
[[@guiligan](https://github.com/guiligan)]
-1. [Azri Solutions](http://www.azrisolutions.com/)
[[@userimack](https://github.com/userimack)]
1. [BandwidthX](http://www.bandwidthx.com)
[[@dineshdsharma](https://github.com/dineshdsharma)]
1. [Bellhops](https://github.com/bellhops)
1. [BelugaDB](https://belugadb.com)
[[@fabio-nukui](https://github.com/fabio-nukui) &
[@joao-sallaberry](http://github.com/joao-sallaberry) &
[@lucianoviola](https://github.com/lucianoviola) &
[@tmatuki](https://github.com/tmatuki)]
1. [BlaBlaCar](https://www.blablacar.com)
[[@puckel](https://github.com/puckel) & [@wmorin](https://github.com/wmorin)]
1. [Bloc](https://www.bloc.io) [[@dpaola2](https://github.com/dpaola2)]
+1. [Blue Yonder](http://www.blue-yonder.com)
[[@blue-yonder](https://github.com/blu
incubator-airflow git commit: [AIRFLOW-2452] Document field_dict must be OrderedDict
Repository: incubator-airflow
Updated Branches:
refs/heads/master 7c233179e -> 648b14b4d
[AIRFLOW-2452] Document field_dict must be OrderedDict
HiveCliHook.load_file has a parameter called
field_dict, which defines name-type pairs
for columns, must be OrderedDict so as to
keep columns' order, but it's undocumented.
This PR adds an note about that, and fixes
HiveCliHook.load_df function which calls
load_file internally.
Closes #3347 from sekikn/AIRFLOW-2452
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/648b14b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/648b14b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/648b14b4
Branch: refs/heads/master
Commit: 648b14b4d95bf3aca26e8b54ffe8585b52efc8fd
Parents: 7c23317
Author: Kengo Seki
Authored: Tue May 15 10:53:29 2018 -0700
Committer: r39132
Committed: Tue May 15 10:53:29 2018 -0700
--
airflow/hooks/hive_hooks.py | 16 +++-
tests/hooks/test_hive_hook.py | 2 ++
2 files changed, 13 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/648b14b4/airflow/hooks/hive_hooks.py
--
diff --git a/airflow/hooks/hive_hooks.py b/airflow/hooks/hive_hooks.py
index 6c9a907..92958c0 100644
--- a/airflow/hooks/hive_hooks.py
+++ b/airflow/hooks/hive_hooks.py
@@ -27,6 +27,7 @@ import itertools
import re
import subprocess
import time
+from collections import OrderedDict
from tempfile import NamedTemporaryFile
import hmsclient
@@ -303,8 +304,9 @@ class HiveCliHook(BaseHook):
:param recreate: whether to drop and recreate the table at every
execution
:type recreate: bool
-:param field_dict: mapping from column name to hive data type
-:type field_dict: dict
+:param field_dict: mapping from column name to hive data type.
+Note that it must be OrderedDict so as to keep columns' order.
+:type field_dict: OrderedDict
:param encoding: string encoding to use when writing DataFrame to file
:type encoding: str
:param pandas_kwargs: passed to DataFrame.to_csv
@@ -325,7 +327,10 @@ class HiveCliHook(BaseHook):
'V': 'STRING'# void
}
-return dict((col, DTYPE_KIND_HIVE_TYPE[dtype.kind]) for col, dtype
in df.dtypes.iteritems())
+d = OrderedDict()
+for col, dtype in df.dtypes.iteritems():
+d[col] = DTYPE_KIND_HIVE_TYPE[dtype.kind]
+return d
if pandas_kwargs is None:
pandas_kwargs = {}
@@ -378,8 +383,9 @@ class HiveCliHook(BaseHook):
:param delimiter: field delimiter in the file
:type delimiter: str
:param field_dict: A dictionary of the fields name in the file
-as keys and their Hive types as values
-:type field_dict: dict
+as keys and their Hive types as values.
+Note that it must be OrderedDict so as to keep columns' order.
+:type field_dict: OrderedDict
:param create: whether to create the table if it doesn't exist
:type create: bool
:param overwrite: whether to overwrite the data in table or partition
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/648b14b4/tests/hooks/test_hive_hook.py
--
diff --git a/tests/hooks/test_hive_hook.py b/tests/hooks/test_hive_hook.py
index 9eda9e3..3c278e3 100644
--- a/tests/hooks/test_hive_hook.py
+++ b/tests/hooks/test_hive_hook.py
@@ -25,6 +25,7 @@ import random
import mock
import unittest
+from collections import OrderedDict
from hmsclient import HMSClient
from airflow.exceptions import AirflowException
@@ -130,6 +131,7 @@ class TestHiveCliHook(unittest.TestCase):
kwargs = mock_load_file.call_args[1]
self.assertEqual(kwargs["delimiter"], delimiter)
self.assertEqual(kwargs["field_dict"], {"c": u"STRING"})
+self.assertTrue(isinstance(kwargs["field_dict"], OrderedDict))
self.assertEqual(kwargs["table"], table)
incubator-airflow git commit: [AIRFLOW-2420] Azure Data Lake Hook
Repository: incubator-airflow
Updated Branches:
refs/heads/master e3bbc314e -> 7c233179e
[AIRFLOW-2420] Azure Data Lake Hook
Add AzureDataLakeHook as a first step to enable
Airflow connect to
Azure Data Lake.
The hook has a simple interface to upload and
download files with all
parameters available in Azure Data Lake sdk and
also a check_for_file
to query if a file exists in data lake.
[AIRFLOW-2420] Add functionality for Azure Data
Lake
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW-242
0) issues and references them in the PR title.
-
https://issues.apache.org/jira/browse/AIRFLOW-2420
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
This PR creates Azure Data Lake hook
(adl_hook.AdlHook) and all the setup required to
create a new Azure Data Lake connection.
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
Adds tests to airflow.hooks.adl_hook.py in
tests.hooks.test_adl_hook.py
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes # from marcusrehm/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7c233179
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7c233179
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7c233179
Branch: refs/heads/master
Commit: 7c233179e91818bd641b283934a73cc84a51ca03
Parents: e3bbc31
Author: Marcus Rehm
Authored: Tue May 15 10:30:54 2018 -0700
Committer: r39132
Committed: Tue May 15 10:30:54 2018 -0700
--
airflow/contrib/hooks/__init__.py | 3 +-
airflow/contrib/hooks/azure_data_lake_hook.py | 141 +++
airflow/models.py | 4 +
airflow/utils/db.py | 6 +-
docs/integration.rst| 19 ++-
setup.py| 16 ++-
.../contrib/hooks/test_azure_data_lake_hook.py | 105 ++
7 files changed, 286 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7c233179/airflow/contrib/hooks/__init__.py
--
diff --git a/airflow/contrib/hooks/__init__.py
b/airflow/contrib/hooks/__init__.py
index 81ce8f0..1c92deb 100644
--- a/airflow/contrib/hooks/__init__.py
+++ b/airflow/contrib/hooks/__init__.py
@@ -58,7 +58,8 @@ _hooks = {
'wasb_hook': ['WasbHook'],
'gcp_pubsub_hook': ['PubSubHook'],
'jenkins_hook': ['JenkinsHook'],
-'aws_dynamodb_hook': ['AwsDynamoDBHook']
+'aws_dynamodb_hook': ['AwsDynamoDBHook'],
+'azure_data_lake_hook': ['AzureDataLakeHook'],
}
import os as _os
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7c233179/airflow/contrib/hooks/azure_data_lake_hook.py
--
diff --git a/airflow/contrib/hooks/azure_data_lake_hook.py
b/airflow/contrib/hooks/azure_data_lake_hook.py
new file mode 100644
index 000..1a02d78
--- /dev/null
+++ b/airflow/contrib/hooks/azure_data_lake_hook.py
@@ -0,0 +1,141 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KI
incubator-airflow git commit: closes apache/incubator-airflow#2291 *Obsolete PR.*
Repository: incubator-airflow Updated Branches: refs/heads/master 992b19bb7 -> e3bbc314e closes apache/incubator-airflow#2291 *Obsolete PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e3bbc314 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e3bbc314 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e3bbc314 Branch: refs/heads/master Commit: e3bbc314ee8e3911eb7c167f48eb0171d8b2c032 Parents: 992b19b Author: r39132 Authored: Tue May 15 10:11:47 2018 -0700 Committer: r39132 Committed: Tue May 15 10:11:47 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2318 *Obsolete PR.*
Repository: incubator-airflow Updated Branches: refs/heads/master f9e6b1fbf -> 992b19bb7 closes apache/incubator-airflow#2318 *Obsolete PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/992b19bb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/992b19bb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/992b19bb Branch: refs/heads/master Commit: 992b19bb79d1acfdaf20332a52b2a9be0a1abbe7 Parents: f9e6b1f Author: r39132 Authored: Tue May 15 10:10:58 2018 -0700 Committer: r39132 Committed: Tue May 15 10:10:58 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2472 *Obsolete PR.*
Repository: incubator-airflow Updated Branches: refs/heads/master eb99401d0 -> f9e6b1fbf closes apache/incubator-airflow#2472 *Obsolete PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f9e6b1fb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f9e6b1fb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f9e6b1fb Branch: refs/heads/master Commit: f9e6b1fbfd1127c921aaccd24a332b5e87570ba5 Parents: eb99401 Author: r39132 Authored: Tue May 15 10:08:38 2018 -0700 Committer: r39132 Committed: Tue May 15 10:08:38 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#3325 *Need to be reopened and follow contributor guidelines*
Repository: incubator-airflow Updated Branches: refs/heads/master 2a55ffe0c -> 06017f66e closes apache/incubator-airflow#3325 *Need to be reopened and follow contributor guidelines* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/06017f66 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/06017f66 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/06017f66 Branch: refs/heads/master Commit: 06017f66eed0aef2c9c441bbd0ae5869059d011e Parents: 2a55ffe Author: r39132 Authored: Wed May 9 14:14:04 2018 -0700 Committer: r39132 Committed: Wed May 9 14:14:04 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-XXX] Add Quantopian to list of Airflow users
Repository: incubator-airflow Updated Branches: refs/heads/master e5f2a38d6 -> e5cdabb16 [AIRFLOW-XXX] Add Quantopian to list of Airflow users We've been using it in production for a few months now, and we love it! :) Closes #3331 from Eronarn/patch-1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e5cdabb1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e5cdabb1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e5cdabb1 Branch: refs/heads/master Commit: e5cdabb166b09f9bad09ab3c0a9774efb96a0a7e Parents: e5f2a38 Author: James Meickle Authored: Tue May 8 11:15:17 2018 -0700 Committer: r39132 Committed: Tue May 8 11:15:21 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e5cdabb1/README.md -- diff --git a/README.md b/README.md index e45c6ff..f09eb2d 100644 --- a/README.md +++ b/README.md @@ -201,6 +201,7 @@ Currently **officially** using Airflow: 1. [Postmates](http://www.postmates.com) [[@syeoryn](https://github.com/syeoryn)] 1. [Pronto Tools](http://www.prontotools.io/) [[@zkan](https://github.com/zkan) & [@mesodiar](https://github.com/mesodiar)] 1. [Qplum](https://qplum.co) [[@manti](https://github.com/manti)] +1. [Quantopian](https://www.quantopian.com/) [[@eronarn](http://github.com/eronarn)] 1. [Qubole](https://qubole.com) [[@msumit](https://github.com/msumit)] 1. [Quizlet](https://quizlet.com) [[@quizlet](https://github.com/quizlet)] 1. [Quora](https://www.quora.com/)
incubator-airflow git commit: [AIRFLOW-1978] Add WinRM windows operator and hook
Repository: incubator-airflow
Updated Branches:
refs/heads/master b18b437c2 -> e5f2a38d6
[AIRFLOW-1978] Add WinRM windows operator and hook
Closes #3316 from cloneluke/winrm_connector2
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e5f2a38d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e5f2a38d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e5f2a38d
Branch: refs/heads/master
Commit: e5f2a38d6aff69bb4d31c792f7e9234b51646675
Parents: b18b437
Author: Luke Bodeen
Authored: Tue May 8 11:12:52 2018 -0700
Committer: r39132
Committed: Tue May 8 11:12:59 2018 -0700
--
.../example_dags/example_winrm_operator.py | 73 ++
airflow/contrib/hooks/__init__.py | 1 +
airflow/contrib/hooks/winrm_hook.py | 137 +++
airflow/contrib/operators/__init__.py | 1 +
airflow/contrib/operators/winrm_operator.py | 110 +++
docs/code.rst | 6 +-
setup.py| 76 +-
7 files changed, 370 insertions(+), 34 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e5f2a38d/airflow/contrib/example_dags/example_winrm_operator.py
--
diff --git a/airflow/contrib/example_dags/example_winrm_operator.py
b/airflow/contrib/example_dags/example_winrm_operator.py
new file mode 100644
index 000..195bf5d
--- /dev/null
+++ b/airflow/contrib/example_dags/example_winrm_operator.py
@@ -0,0 +1,73 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Written By: Ekhtiar Syed
+# Last Update: 8th April 2016
+# Caveat: This Dag will not run because of missing scripts.
+# The purpose of this is to give you a sample of a real world example DAG!
+#
+
+#
+# Load The Dependencies
+#
+import airflow
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.models import DAG
+from datetime import timedelta
+
+from airflow.contrib.hooks import WinRMHook
+from airflow.contrib.operators.winrm_operator import WinRMOperator
+
+
+args = {
+'owner': 'airflow',
+'start_date': airflow.utils.dates.days_ago(2)
+}
+
+dag = DAG(
+dag_id='POC_winrm_parallel', default_args=args,
+schedule_interval='0 0 * * *',
+dagrun_timeout=timedelta(minutes=60))
+
+cmd = 'ls -l'
+run_this_last = DummyOperator(task_id='run_this_last', dag=dag)
+
+winRMHook = WinRMHook(ssh_conn_id='ssh_POC1')
+
+t1 = WinRMOperator(
+task_id="wintask1",
+command='ls -altr',
+winrm_hook=winRMHook,
+dag=dag)
+
+t2 = WinRMOperator(
+task_id="wintask2",
+command='sleep 60',
+winrm_hook=winRMHook,
+dag=dag)
+
+t3 = WinRMOperator(
+task_id="wintask3",
+command='echo \'luke test\' ',
+winrm_hook=winRMHook,
+dag=dag)
+
+t1.set_downstream(run_this_last)
+t2.set_downstream(run_this_last)
+t3.set_downstream(run_this_last)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e5f2a38d/airflow/contrib/hooks/__init__.py
--
diff --git a/airflow/contrib/hooks/__init__.py
b/airflow/contrib/hooks/__init__.py
index 7f9d59e..81ce8f0 100644
--- a/airflow/contrib/hooks/__init__.py
+++ b/airflow/contrib/hooks/__init__.py
@@ -42,6 +42,7 @@ _hooks = {
'ftps_hook': ['FTPSHook'],
'vertica_hook': ['VerticaHook'],
'ssh_hook': ['SSHHook'],
+'winrm_hook': ['WinRMHook'],
'sftp_hook': ['SFTPHook'],
'bigquery_hook': ['BigQueryHook'],
'qubole_hook': ['QuboleHook'],
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e
[3/4] incubator-airflow git commit: closes apache/incubator-airflow#2770 *Closed for inactivity.*
closes apache/incubator-airflow#2770 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f9eda3d4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f9eda3d4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f9eda3d4 Branch: refs/heads/master Commit: f9eda3d49236e8a4ea910f43c23e0955fe6cf727 Parents: 1610d84 Author: r39132 Authored: Mon May 7 11:44:45 2018 -0700 Committer: r39132 Committed: Mon May 7 11:44:45 2018 -0700 -- --
[4/4] incubator-airflow git commit: closes apache/incubator-airflow#2768 *Closed for inactivity.*
closes apache/incubator-airflow#2768 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c27d8fd0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c27d8fd0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c27d8fd0 Branch: refs/heads/master Commit: c27d8fd0b03c9fb1ccb01132c3078d18cf25695c Parents: f9eda3d Author: r39132 Authored: Mon May 7 11:44:48 2018 -0700 Committer: r39132 Committed: Mon May 7 11:44:48 2018 -0700 -- --
[1/4] incubator-airflow git commit: closes apache/incubator-airflow#2772 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master 214583996 -> c27d8fd0b closes apache/incubator-airflow#2772 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/788625a5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/788625a5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/788625a5 Branch: refs/heads/master Commit: 788625a58368c92517f2566ff85bf78b71ba0b60 Parents: 2145839 Author: r39132 Authored: Mon May 7 11:44:40 2018 -0700 Committer: r39132 Committed: Mon May 7 11:44:40 2018 -0700 -- --
[2/4] incubator-airflow git commit: closes apache/incubator-airflow#2769 *Closed for inactivity.*
closes apache/incubator-airflow#2769 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/1610d844 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/1610d844 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/1610d844 Branch: refs/heads/master Commit: 1610d8443eadadac9d78416c73173c832c31ba8e Parents: 788625a Author: r39132 Authored: Mon May 7 11:44:42 2018 -0700 Committer: r39132 Committed: Mon May 7 11:44:42 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1952] Add the navigation bar color parameter
Repository: incubator-airflow
Updated Branches:
refs/heads/master fcf8435f4 -> 214583996
[AIRFLOW-1952] Add the navigation bar color parameter
ENH: Add the navigation bar color parameter
When operating multiple Airflow's (eg. Production,
Staging, etc.),
we cannot distinguish which Airflow is.
This feature enables us to distinguish which
Airflow you see
by the color of navigation bar.
Merge branch 'master' into add-navigation-bar-
color-parameter
Closes #2903 from Licht-T/add-navigation-bar-
color-parameter
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/21458399
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/21458399
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/21458399
Branch: refs/heads/master
Commit: 21458399609a879b318ddbb37b52958c62baadc2
Parents: fcf8435
Author: Licht-T
Authored: Mon May 7 11:18:32 2018 -0700
Committer: r39132
Committed: Mon May 7 11:18:32 2018 -0700
--
airflow/config_templates/default_airflow.cfg | 3 +++
airflow/www/app.py | 1 +
airflow/www/templates/admin/master.html | 2 +-
3 files changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/21458399/airflow/config_templates/default_airflow.cfg
--
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 6da4287..b91961e 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -277,6 +277,9 @@ page_size = 100
# Use FAB-based webserver with RBAC feature
rbac = False
+# Define the color of navigation bar
+navbar_color = #007A87
+
[email]
email_backend = airflow.utils.email.send_email_smtp
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/21458399/airflow/www/app.py
--
diff --git a/airflow/www/app.py b/airflow/www/app.py
index ca0589d..e9b101d 100644
--- a/airflow/www/app.py
+++ b/airflow/www/app.py
@@ -156,6 +156,7 @@ def create_app(config=None, testing=False):
def jinja_globals():
return {
'hostname': get_hostname(),
+'navbar_color': configuration.get('webserver', 'NAVBAR_COLOR'),
}
@app.teardown_appcontext
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/21458399/airflow/www/templates/admin/master.html
--
diff --git a/airflow/www/templates/admin/master.html
b/airflow/www/templates/admin/master.html
index 1f0020e..cb1e9f1 100644
--- a/airflow/www/templates/admin/master.html
+++ b/airflow/www/templates/admin/master.html
@@ -51,7 +51,7 @@
{% block page_body %}
-
+
incubator-airflow git commit: closes apache/incubator-airflow#3237 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master 69da86658 -> fcf8435f4 closes apache/incubator-airflow#3237 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fcf8435f Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fcf8435f Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fcf8435f Branch: refs/heads/master Commit: fcf8435f4baddf5a1ef95d3f894437f37a9309a7 Parents: 69da866 Author: r39132 Authored: Mon May 7 11:14:41 2018 -0700 Committer: r39132 Committed: Mon May 7 11:14:41 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-XXX] Update README.md with Craig@Work
Repository: incubator-airflow Updated Branches: refs/heads/master 09cdd4aa0 -> 29ae02a07 [AIRFLOW-XXX] Update README.md with Craig@Work Closes #3311 from allanjsx/patch-1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/29ae02a0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/29ae02a0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/29ae02a0 Branch: refs/heads/master Commit: 29ae02a070132543ac92706d74d9a5dc676053d9 Parents: 09cdd4a Author: Allan Jiang Authored: Sat May 5 10:27:43 2018 -0700 Committer: r39132 Committed: Sat May 5 10:27:49 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/29ae02a0/README.md -- diff --git a/README.md b/README.md index 07371ab..e45c6ff 100644 --- a/README.md +++ b/README.md @@ -120,6 +120,7 @@ Currently **officially** using Airflow: 1. [Chartboost](https://www.chartboost.com) [[@cgelman](https://github.com/cgelman) & [@dclubb](https://github.com/dclubb)] 1. [ContaAzul](https://www.contaazul.com) [[@bern4rdelli](https://github.com/bern4rdelli), [@renanleme](https://github.com/renanleme) & [@sabino](https://github.com/sabino)] 1. [Cotap](https://github.com/cotap/) [[@maraca](https://github.com/maraca) & [@richardchew](https://github.com/richardchew)] +1. [Craig@Work](https://www.craigatwork.com) 1. [CreditCards.com](https://www.creditcards.com/)[[@vmAggies](https://github.com/vmAggies) & [@jay-wallaby](https://github.com/jay-wallaby)] 1. [Credit Karma](https://www.creditkarma.com/) [[@preete-dixit-ck](https://github.com/preete-dixit-ck) & [@harish-gaggar-ck](https://github.com/harish-gaggar-ck) & [@greg-finley-ck](https://github.com/greg-finley-ck)] 1. [Creditas](https://www.creditas.com.br) [[@dcassiano](https://github.com/dcassiano)]
incubator-airflow git commit: closes apache/incubator-airflow#2923 *Closed in favor of newer PR*
Repository: incubator-airflow Updated Branches: refs/heads/master 16bae5634 -> 09cdd4aa0 closes apache/incubator-airflow#2923 *Closed in favor of newer PR* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/09cdd4aa Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/09cdd4aa Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/09cdd4aa Branch: refs/heads/master Commit: 09cdd4aa0b47a60eb30f11a6592210f569494ab1 Parents: 16bae56 Author: r39132 Authored: Fri May 4 16:46:49 2018 -0700 Committer: r39132 Committed: Fri May 4 16:46:49 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2478 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master 97ab9e762 -> d1f7af393 closes apache/incubator-airflow#2478 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d1f7af39 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d1f7af39 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d1f7af39 Branch: refs/heads/master Commit: d1f7af393f2d0e27807d0fc31d8023544f5c2aac Parents: 97ab9e7 Author: r39132 Authored: Thu May 3 23:02:59 2018 -0700 Committer: r39132 Committed: Thu May 3 23:02:59 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1812] Update logging example
Repository: incubator-airflow
Updated Branches:
refs/heads/master 3a28ceb5c -> 97ab9e762
[AIRFLOW-1812] Update logging example
The logging has changed, therefore we should also
update the
updating.md guide
Closes #2784 from Fokko/AIRFLOW-1812-update-
logging-example
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/97ab9e76
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/97ab9e76
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/97ab9e76
Branch: refs/heads/master
Commit: 97ab9e762c1f325317fd8003e2846e931137a1e6
Parents: 3a28ceb
Author: Fokko Driesprong
Authored: Thu May 3 22:58:44 2018 -0700
Committer: r39132
Committed: Thu May 3 22:58:44 2018 -0700
--
UPDATING.md | 115 +--
1 file changed, 112 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/97ab9e76/UPDATING.md
--
diff --git a/UPDATING.md b/UPDATING.md
index 8006876..defd95b 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -61,9 +61,9 @@ Dataflow job labeling is now supported in
Dataflow{Java,Python}Operator with a d
to 2.2.0 or greater.
### Redshift to S3 Operator
-With Airflow 1.9 or lower, Unload operation always included header row. In
order to include header row,
+With Airflow 1.9 or lower, Unload operation always included header row. In
order to include header row,
we need to turn off parallel unload. It is preferred to perform unload
operation using all nodes so that it is
-faster for larger tables. So, parameter called `include_header` is added and
default is set to False.
+faster for larger tables. So, parameter called `include_header` is added and
default is set to False.
Header row will be added only if this parameter is set True and also in that
case parallel will be automatically turned off (`PARALLEL OFF`)
### Google cloud connection string
@@ -119,7 +119,116 @@ logging_config_class =
my.path.default_local_settings.LOGGING_CONFIG
The logging configuration file needs to be on the `PYTHONPATH`, for example
`$AIRFLOW_HOME/config`. This directory is loaded by default. Any directory may
be added to the `PYTHONPATH`, this might be handy when the config is in another
directory or a volume is mounted in case of Docker.
-The config can be taken from
`airflow/config_templates/airflow_local_settings.py` as a starting point. Copy
the contents to `${AIRFLOW_HOME}/config/airflow_local_settings.py`, and alter
the config as is preferred.
+The config can be taken from
`airflow/config_templates/airflow_local_settings.py` as a starting point. Copy
the contents to `${AIRFLOW_HOME}/config/airflow_local_settings.py`, and alter
the config as is preferred.
+
+```
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+from airflow import configuration as conf
+
+# TODO: Logging format and level should be configured
+# in this file instead of from airflow.cfg. Currently
+# there are other log format and level configurations in
+# settings.py and cli.py. Please see AIRFLOW-1455.
+
+LOG_LEVEL = conf.get('core', 'LOGGING_LEVEL').upper()
+LOG_FORMAT = conf.get('core', 'log_format')
+
+BASE_LOG_FOLDER = conf.get('core', 'BASE_LOG_FOLDER')
+PROCESSOR_LOG_FOLDER = conf.get('scheduler', 'child_process_log_directory')
+
+FILENAME_TEMPLATE = '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number
}}.log'
+PROCESSOR_FILENAME_TEMPLATE = '{{ filename }}.log'
+
+DEFAULT_LOGGING_CONFIG = {
+'version': 1,
+'disable_existing_loggers': False,
+'formatters': {
+'airflow.task': {
+'format': LOG_FORMAT,
+},
+'airflow.processor': {
+'format': LOG_FORMAT,
+},
+},
+'handlers': {
+'console': {
+'class': 'logging.StreamHandler',
+'formatter': 'airflow.task',
+'stream': 'ext://sys.stdout'
+},
+'file.task': {
+'class': 'airflow.utils.log.file_task_handler.FileTaskHandler',
+'formatter': 'airflow.task',
+'base_log_folder': os.path.expanduser(BASE_LOG_FOLDER),
+'filename_template': FILENAME_TEMPLATE,
+
[2/2] incubator-airflow git commit: closes apache/incubator-airflow#2675 *Closed for inactivity.*
closes apache/incubator-airflow#2675 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3a28ceb5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3a28ceb5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3a28ceb5 Branch: refs/heads/master Commit: 3a28ceb5c5c5d48e2cded54a010b6f3ca45ac84e Parents: 02a88f6 Author: r39132 Authored: Thu May 3 22:52:39 2018 -0700 Committer: r39132 Committed: Thu May 3 22:52:39 2018 -0700 -- --
[1/2] incubator-airflow git commit: closes apache/incubator-airflow#2962 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master b02820a7a -> 3a28ceb5c closes apache/incubator-airflow#2962 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/02a88f6f Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/02a88f6f Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/02a88f6f Branch: refs/heads/master Commit: 02a88f6f6ff34b07630efe6157e1fec3bc3bb505 Parents: b02820a Author: r39132 Authored: Thu May 3 22:51:30 2018 -0700 Committer: r39132 Committed: Thu May 3 22:51:30 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-XXX] Fix wrong table header in scheduler.rst
Repository: incubator-airflow Updated Branches: refs/heads/master 2a079b953 -> 6f6884641 [AIRFLOW-XXX] Fix wrong table header in scheduler.rst Closes #3306 from sekikn/table_header Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6f688464 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6f688464 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6f688464 Branch: refs/heads/master Commit: 6f6884641f6eca799f35190ef133329a00a1daaf Parents: 2a079b9 Author: Kengo Seki Authored: Wed May 2 23:47:33 2018 -0700 Committer: r39132 Committed: Wed May 2 23:47:33 2018 -0700 -- docs/scheduler.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6f688464/docs/scheduler.rst -- diff --git a/docs/scheduler.rst b/docs/scheduler.rst index 7e4e544..dfa0a42 100644 --- a/docs/scheduler.rst +++ b/docs/scheduler.rst @@ -45,7 +45,7 @@ a ``str``, or a ``datetime.timedelta`` object. Alternatively, you can also use one of these cron "preset": +--++---+ -| preset | Run once a year at midnight of January 1 | cron | +| preset | meaning | cron | +==++===+ | ``None`` | Don't schedule, use for exclusively "externally triggered" | | | | DAGs | |
incubator-airflow git commit: [AIRFLOW-2406] Add Apache2 License Shield to Readme
Repository: incubator-airflow Updated Branches: refs/heads/master 9a0c4e6ca -> 96d00da5b [AIRFLOW-2406] Add Apache2 License Shield to Readme Closes #3290 from r39132/add_apache2_license_shield_to_readme Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/96d00da5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/96d00da5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/96d00da5 Branch: refs/heads/master Commit: 96d00da5b038d12b01aa538f0f49722435bedbac Parents: 9a0c4e6 Author: r39132 Authored: Tue May 1 23:36:59 2018 -0700 Committer: r39132 Committed: Tue May 1 23:36:59 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/96d00da5/README.md -- diff --git a/README.md b/README.md index 86b4a8c..bdbfbf2 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ [](https://travis-ci.org/apache/incubator-airflow) [](https://codecov.io/github/apache/incubator-airflow?branch=master) [](https://airflow.readthedocs.io/en/latest/?badge=latest) +[](http://www.apache.org/licenses/LICENSE-2.0.txt) [](https://gitter.im/apache/incubator-airflow?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) _NOTE: The transition from 1.8.0 (or before) to 1.8.1 (or after) requires uninstalling Airflow before installing the new version. The package name was changed from `airflow` to `apache-airflow` as of version 1.8.1._
incubator-airflow git commit: closes apache/incubator-airflow#2337 *No longer a bug*
Repository: incubator-airflow Updated Branches: refs/heads/master fef7d304d -> 8c49e8d9b closes apache/incubator-airflow#2337 *No longer a bug* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8c49e8d9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8c49e8d9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8c49e8d9 Branch: refs/heads/master Commit: 8c49e8d9bcd905dcd7a40d467bac94a687d85d75 Parents: fef7d30 Author: r39132 Authored: Tue May 1 22:41:32 2018 -0700 Committer: r39132 Committed: Tue May 1 22:41:32 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2401] Document the use of variables in Jinja template
Repository: incubator-airflow
Updated Branches:
refs/heads/master 8921e7d70 -> a67c13e44
[AIRFLOW-2401] Document the use of variables in Jinja template
Closes #2847 from moe-nadal-ck/patch-1
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a67c13e4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a67c13e4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a67c13e4
Branch: refs/heads/master
Commit: a67c13e44cb96acacc79896b327bbaaae19707a1
Parents: 8921e7d
Author: Moe Nadal
Authored: Mon Apr 30 15:05:47 2018 -0700
Committer: r39132
Committed: Mon Apr 30 15:06:10 2018 -0700
--
docs/concepts.rst | 12
1 file changed, 12 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a67c13e4/docs/concepts.rst
--
diff --git a/docs/concepts.rst b/docs/concepts.rst
index 89c25fe..c28b10f 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -427,6 +427,18 @@ The second call assumes ``json`` content and will be
deserialized into
``bar``. Note that ``Variable`` is a sqlalchemy model and can be used
as such.
+You can use a variable from a jinja template with the syntax :
+
+.. code:: bash
+
+echo {{ var.value. }}
+
+or if you need to deserialize a json object from the variable :
+
+.. code:: bash
+
+echo {{ var.json. }}
+
Branching
=
incubator-airflow git commit: closes apache/incubator-airflow#2814 *Messed up PR - hundreds of old commits.*
Repository: incubator-airflow Updated Branches: refs/heads/master f63a2b1a1 -> 8921e7d70 closes apache/incubator-airflow#2814 *Messed up PR - hundreds of old commits.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8921e7d7 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8921e7d7 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8921e7d7 Branch: refs/heads/master Commit: 8921e7d70c46027c5a3f4b686f70d893ec91eb89 Parents: f63a2b1 Author: r39132 Authored: Mon Apr 30 13:58:40 2018 -0700 Committer: r39132 Committed: Mon Apr 30 13:58:40 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2403] Fix License Headers
Repository: incubator-airflow Updated Branches: refs/heads/master a7e8f4896 -> f63a2b1a1 [AIRFLOW-2403] Fix License Headers Closes #3285 from r39132/fix_license_headers Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f63a2b1a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f63a2b1a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f63a2b1a Branch: refs/heads/master Commit: f63a2b1a1f1c216e22d6c46ecbd56199a2ff398e Parents: a7e8f48 Author: r39132 Authored: Mon Apr 30 13:12:51 2018 -0700 Committer: r39132 Committed: Mon Apr 30 13:12:51 2018 -0700 -- airflow/contrib/hooks/aws_firehose_hook.py| 24 - tests/contrib/hooks/test_aws_firehose_hook.py | 25 -- 2 files changed, 28 insertions(+), 21 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f63a2b1a/airflow/contrib/hooks/aws_firehose_hook.py -- diff --git a/airflow/contrib/hooks/aws_firehose_hook.py b/airflow/contrib/hooks/aws_firehose_hook.py index cf7b2fc..b273e22 100644 --- a/airflow/contrib/hooks/aws_firehose_hook.py +++ b/airflow/contrib/hooks/aws_firehose_hook.py @@ -1,16 +1,20 @@ -# -*- coding: utf-8 -*- # -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. from airflow.contrib.hooks.aws_hook import AwsHook http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f63a2b1a/tests/contrib/hooks/test_aws_firehose_hook.py -- diff --git a/tests/contrib/hooks/test_aws_firehose_hook.py b/tests/contrib/hooks/test_aws_firehose_hook.py index 0a2c809..f22bcde 100644 --- a/tests/contrib/hooks/test_aws_firehose_hook.py +++ b/tests/contrib/hooks/test_aws_firehose_hook.py @@ -1,17 +1,20 @@ -# -*- coding: utf-8 -*- # -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# http://www.apache.org/licenses/LICENSE-2.0 # +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. import unittest import uuid
incubator-airflow git commit: [AIRFLOW-1313] Fix license header
Repository: incubator-airflow Updated Branches: refs/heads/master 9a8e4f775 -> a7e8f4896 [AIRFLOW-1313] Fix license header Closes #3281 from juise/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a7e8f489 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a7e8f489 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a7e8f489 Branch: refs/heads/master Commit: a7e8f48963e38ed233a84b1439715fdbcb9ec619 Parents: 9a8e4f7 Author: Alexander Petrovsky Authored: Mon Apr 30 09:11:22 2018 -0700 Committer: r39132 Committed: Mon Apr 30 09:11:22 2018 -0700 -- airflow/contrib/operators/vertica_to_mysql.py | 23 +++ .../contrib/operators/test_vertica_to_mysql.py | 24 2 files changed, 28 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a7e8f489/airflow/contrib/operators/vertica_to_mysql.py -- diff --git a/airflow/contrib/operators/vertica_to_mysql.py b/airflow/contrib/operators/vertica_to_mysql.py index 9b14e6c..2e9325d 100644 --- a/airflow/contrib/operators/vertica_to_mysql.py +++ b/airflow/contrib/operators/vertica_to_mysql.py @@ -1,16 +1,21 @@ # -*- coding: utf-8 -*- # -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. import logging http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a7e8f489/tests/contrib/operators/test_vertica_to_mysql.py -- diff --git a/tests/contrib/operators/test_vertica_to_mysql.py b/tests/contrib/operators/test_vertica_to_mysql.py index 5c05e3d..615f111 100644 --- a/tests/contrib/operators/test_vertica_to_mysql.py +++ b/tests/contrib/operators/test_vertica_to_mysql.py @@ -1,17 +1,21 @@ # -*- coding: utf-8 -*- # -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. +# http://www.apache.org/licenses/LICENSE-2.0 # +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. import datetime
incubator-airflow git commit: [AIRFLOW-2398] Add BounceX to list of current airflow users
Repository: incubator-airflow Updated Branches: refs/heads/master 19b390128 -> 9a8e4f775 [AIRFLOW-2398] Add BounceX to list of current airflow users Closes #3282 from JoshFerge/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9a8e4f77 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9a8e4f77 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9a8e4f77 Branch: refs/heads/master Commit: 9a8e4f77568783b95efb35b483e1eac10f021ea9 Parents: 19b3901 Author: Josh Ferge Authored: Mon Apr 30 07:03:33 2018 -0700 Committer: r39132 Committed: Mon Apr 30 07:03:37 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9a8e4f77/README.md -- diff --git a/README.md b/README.md index b504cbd..bd09e9c 100644 --- a/README.md +++ b/README.md @@ -105,6 +105,7 @@ Currently **officially** using Airflow: 1. [Boda Telecom Suite - CE](https://github.com/bodastage/bts-ce) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)] 1. [Bodastage Solutions](http://bodastage.com) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)] 1. [Bonnier Broadcasting](http://www.bonnierbroadcasting.com) [[@wileeam](https://github.com/wileeam)] +1. [BounceX](http://www.bouncex.com) [[@JoshFerge](https://github.com/JoshFerge), [@hudsonrio](https://github.com/hudsonrio), [@ronniekritou](https://github.com/ronniekritou)] 1. [California Data Collaborative](https://github.com/California-Data-Collaborative) powered by [ARGO Labs](http://www.argolabs.org) 1. [Carbonite](https://www.carbonite.com) [[@ajbosco](https://github.com/ajbosco)] 1. [Celect](http://www.celect.com) [[@superdosh](https://github.com/superdosh) & [@chadcelect](https://github.com/chadcelect)]
incubator-airflow git commit: closes apache/incubator-airflow#2970 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master 06e90f496 -> e29562e33 closes apache/incubator-airflow#2970 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e29562e3 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e29562e3 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e29562e3 Branch: refs/heads/master Commit: e29562e33a05c66bc640c56143f3a2ca4e0a0b98 Parents: 06e90f4 Author: r39132 Authored: Sun Apr 29 20:20:30 2018 -0700 Committer: r39132 Committed: Sun Apr 29 20:20:30 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2047 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master 8c8d1407f -> 06e90f496 closes apache/incubator-airflow#2047 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/06e90f49 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/06e90f49 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/06e90f49 Branch: refs/heads/master Commit: 06e90f49615896ce5c0f720f0e760c10762e8f4c Parents: 8c8d140 Author: r39132 Authored: Sun Apr 29 20:18:18 2018 -0700 Committer: r39132 Committed: Sun Apr 29 20:18:18 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2539 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master ad28decc7 -> 8c8d1407f closes apache/incubator-airflow#2539 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8c8d1407 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8c8d1407 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8c8d1407 Branch: refs/heads/master Commit: 8c8d1407f8fb374f1f6d8f7788f880461519a430 Parents: ad28dec Author: r39132 Authored: Sun Apr 29 20:16:51 2018 -0700 Committer: r39132 Committed: Sun Apr 29 20:16:51 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1933] Fix some typos
Repository: incubator-airflow Updated Branches: refs/heads/master 3952e050a -> ad28decc7 [AIRFLOW-1933] Fix some typos Closes #2474 from Philippus/patch-1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ad28decc Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ad28decc Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ad28decc Branch: refs/heads/master Commit: ad28decc74a87f6f9fbff57f200199860a2bfa81 Parents: 3952e05 Author: Philippus Baalman Authored: Sun Apr 29 20:08:48 2018 -0700 Committer: r39132 Committed: Sun Apr 29 20:08:48 2018 -0700 -- UPDATING.md | 22 +++--- 1 file changed, 11 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ad28decc/UPDATING.md -- diff --git a/UPDATING.md b/UPDATING.md index 609c8db..8006876 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -74,17 +74,17 @@ With Airflow 1.9 or lower, there were two connection strings for the Google Clou ### SSH Hook updates, along with new SSH Operator & SFTP Operator -SSH Hook now uses Paramiko library to create ssh client connection, instead of sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible. +SSH Hook now uses the Paramiko library to create an ssh client connection, instead of the sub-process based ssh command execution previously (<1.9.0), so this is backward incompatible. - update SSHHook constructor - - use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer test_ssh_operator.py for usage info. - - SFTPOperator is added to perform secure file transfer from serverA to serverB. Refer test_sftp_operator.py.py for usage info. + - use SSHOperator class in place of SSHExecuteOperator which is removed now. Refer to test_ssh_operator.py for usage info. + - SFTPOperator is added to perform secure file transfer from serverA to serverB. Refer to test_sftp_operator.py.py for usage info. - No updates are required if you are using ftpHook, it will continue to work as is. ### S3Hook switched to use Boto3 The airflow.hooks.S3_hook.S3Hook has been switched to use boto3 instead of the older boto (a.k.a. boto2). This results in a few backwards incompatible changes to the following classes: S3Hook: - the constructors no longer accepts `s3_conn_id`. It is now called `aws_conn_id`. - - the default conneciton is now "aws_default" instead of "s3_default" + - the default connection is now "aws_default" instead of "s3_default" - the return type of objects returned by `get_bucket` is now boto3.s3.Bucket - the return type of `get_key`, and `get_wildcard_key` is now an boto3.S3.Object. @@ -106,7 +106,7 @@ Once a logger has determined that a message needs to be processed, it is passed Changes in Airflow Logging -Airflow's logging mechanism has been refactored to uses Pythonâs builtin `logging` module to perform logging of the application. By extending classes with the existing `LoggingMixin`, all the logging will go through a central logger. Also the `BaseHook` and `BaseOperator` already extends this class, so it is easily available to do logging. +Airflow's logging mechanism has been refactored to use Pythonâs builtin `logging` module to perform logging of the application. By extending classes with the existing `LoggingMixin`, all the logging will go through a central logger. Also the `BaseHook` and `BaseOperator` already extend this class, so it is easily available to do logging. The main benefit is easier configuration of the logging by setting a single centralized python file. Disclaimer; there is still some inline configuration, but this will be removed eventually. The new logging class is defined by setting the dotted classpath in your `~/airflow/airflow.cfg` file: @@ -153,7 +153,7 @@ The `file_task_handler` logger has been made more flexible. The default format c If you are logging to Google cloud storage, please see the [Google cloud platform documentation](https://airflow.incubator.apache.org/integration.html#gcp-google-cloud-platform) for logging instructions. If you are using S3, the instructions should be largely the same as the Google cloud platform instructions above. You will need a custom logging config. The `REMOTE_BASE_LOG_FOLDER` configuration key in your airflow config has been removed, therefore you will need to take the following steps: - - Copy the logging configuration from [`airflow/config_templates/airflow_logging_settings.py`](https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/airflow_local_settings.py) and copy it. + - Copy the logging configuration
incubator-airflow git commit: closes apache/incubator-airflow#2930 *Fix belongs in SQLAlchemy.*
Repository: incubator-airflow Updated Branches: refs/heads/master 72f15a108 -> 3952e050a closes apache/incubator-airflow#2930 *Fix belongs in SQLAlchemy.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3952e050 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3952e050 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3952e050 Branch: refs/heads/master Commit: 3952e050a1fe6191f031c2af92785b1823e8f3e3 Parents: 72f15a1 Author: r39132 Authored: Sun Apr 29 10:16:30 2018 -0700 Committer: r39132 Committed: Sun Apr 29 10:16:30 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1313] Add vertica_to_mysql operator
Repository: incubator-airflow
Updated Branches:
refs/heads/master c5d35761f -> c3aa8e31f
[AIRFLOW-1313] Add vertica_to_mysql operator
Closes #2370 from juise/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c3aa8e31
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c3aa8e31
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c3aa8e31
Branch: refs/heads/master
Commit: c3aa8e31fae4edeff8abe6d38e573ef16583496c
Parents: c5d3576
Author: Alexander Petrovsky
Authored: Sun Apr 29 00:05:23 2018 -0700
Committer: r39132
Committed: Sun Apr 29 00:05:23 2018 -0700
--
airflow/contrib/operators/vertica_to_mysql.py | 142 +++
scripts/ci/requirements.txt | 1 +
.../contrib/operators/test_vertica_to_mysql.py | 87
3 files changed, 230 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c3aa8e31/airflow/contrib/operators/vertica_to_mysql.py
--
diff --git a/airflow/contrib/operators/vertica_to_mysql.py
b/airflow/contrib/operators/vertica_to_mysql.py
new file mode 100644
index 000..9b14e6c
--- /dev/null
+++ b/airflow/contrib/operators/vertica_to_mysql.py
@@ -0,0 +1,142 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from airflow.contrib.hooks.vertica_hook import VerticaHook
+from airflow.hooks.mysql_hook import MySqlHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+
+from contextlib import closing
+
+import unicodecsv as csv
+from tempfile import NamedTemporaryFile
+
+
+class VerticaToMySqlTransfer(BaseOperator):
+"""
+Moves data from Vertica to MySQL.
+
+:param sql: SQL query to execute against the Vertica database
+:type sql: str
+:param vertica_conn_id: source Vertica connection
+:type vertica_conn_id: str
+:param mysql_table: target MySQL table, use dot notation to target a
+specific database
+:type mysql_table: str
+:param mysql_conn_id: source mysql connection
+:type mysql_conn_id: str
+:param mysql_preoperator: sql statement to run against MySQL prior to
+import, typically use to truncate of delete in place of the data
+coming in, allowing the task to be idempotent (running the task
+twice won't double load data)
+:type mysql_preoperator: str
+:param mysql_postoperator: sql statement to run against MySQL after the
+import, typically used to move data from staging to production
+and issue cleanup commands.
+:type mysql_postoperator: str
+:param bulk_load: flag to use bulk_load option. This loads MySQL directly
+from a tab-delimited text file using the LOAD DATA LOCAL INFILE
command.
+This option requires an extra connection parameter for the
+destination MySQL connection: {'local_infile': true}.
+:type bulk_load: bool
+"""
+
+template_fields = ('sql', 'mysql_table', 'mysql_preoperator',
+'mysql_postoperator')
+template_ext = ('.sql',)
+ui_color = '#a0e08c'
+
+@apply_defaults
+def __init__(
+self,
+sql,
+mysql_table,
+vertica_conn_id='vertica_default',
+mysql_conn_id='mysql_default',
+mysql_preoperator=None,
+mysql_postoperator=None,
+bulk_load=False,
+*args, **kwargs):
+super(VerticaToMySqlTransfer, self).__init__(*args, **kwargs)
+self.sql = sql
+self.mysql_table = mysql_table
+self.mysql_conn_id = mysql_conn_id
+self.mysql_preoperator = mysql_preoperator
+self.mysql_postoperator = mysql_postoperator
+self.vertica_conn_id = vertica_conn_id
+self.bulk_load = bulk_load
+
+def execute(self, context):
+vertica = VerticaHook(vertica_conn_id=self.vertica_conn_id)
+mysql = MySqlHook(mysql_conn_id=self.mysql_conn_id)
+
+tmpfile = None
+result = None
+
+selected_columns = []
+
+count = 0
+with closing(vertica.get_conn()) as conn:
+with closing(conn.cursor()) as cursor:
+cursor.execut
incubator-airflow git commit: closes apache/incubator-airflow#3209 *PR in heavy need of squashing and cleanup.*
Repository: incubator-airflow Updated Branches: refs/heads/master b34c29635 -> c5d35761f closes apache/incubator-airflow#3209 *PR in heavy need of squashing and cleanup.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c5d35761 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c5d35761 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c5d35761 Branch: refs/heads/master Commit: c5d35761f4768f434527b3a26b101b8d87f7965f Parents: b34c296 Author: r39132 Authored: Sat Apr 28 23:54:19 2018 -0700 Committer: r39132 Committed: Sat Apr 28 23:54:19 2018 -0700 -- --
[1/2] incubator-airflow git commit: closes apache/incubator-airflow#2555 *Fixed by another PR.*
Repository: incubator-airflow Updated Branches: refs/heads/master f67e9673f -> b34c29635 closes apache/incubator-airflow#2555 *Fixed by another PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/99cf26df Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/99cf26df Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/99cf26df Branch: refs/heads/master Commit: 99cf26dfea575f5bd2d0e2e764416259d34c8a41 Parents: f67e967 Author: r39132 Authored: Sat Apr 28 23:35:10 2018 -0700 Committer: r39132 Committed: Sat Apr 28 23:35:10 2018 -0700 -- --
[2/2] incubator-airflow git commit: closes apache/incubator-airflow#2555 *Fixed by another PR.*
closes apache/incubator-airflow#2555 *Fixed by another PR.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b34c2963 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b34c2963 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b34c2963 Branch: refs/heads/master Commit: b34c29635065a6acee14851758aecd4b4f4e9dfd Parents: 99cf26d Author: r39132 Authored: Sat Apr 28 23:35:19 2018 -0700 Committer: r39132 Committed: Sat Apr 28 23:35:19 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2744 *Closed for inactivity.*
Repository: incubator-airflow Updated Branches: refs/heads/master 2d588e943 -> f67e9673f closes apache/incubator-airflow#2744 *Closed for inactivity.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f67e9673 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f67e9673 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f67e9673 Branch: refs/heads/master Commit: f67e9673f5ced64aa3681cf9343a4b8b73f7903b Parents: 2d588e9 Author: r39132 Authored: Sat Apr 28 23:30:32 2018 -0700 Committer: r39132 Committed: Sat Apr 28 23:30:32 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1575] Add AWS Kinesis Firehose Hook for inserting batch records
Repository: incubator-airflow
Updated Branches:
refs/heads/master e691acc11 -> 2d588e943
[AIRFLOW-1575] Add AWS Kinesis Firehose Hook for inserting batch records
Closes #3275 from sid88in/feature/kinesis_hookv2
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2d588e94
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2d588e94
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2d588e94
Branch: refs/heads/master
Commit: 2d588e9433cd9a1a1381cf939f579f7d7e53330f
Parents: e691acc
Author: sid.gupta
Authored: Sat Apr 28 23:11:36 2018 -0700
Committer: r39132
Committed: Sat Apr 28 23:11:36 2018 -0700
--
airflow/contrib/hooks/aws_firehose_hook.py| 52
tests/contrib/hooks/test_aws_firehose_hook.py | 70 ++
2 files changed, 122 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2d588e94/airflow/contrib/hooks/aws_firehose_hook.py
--
diff --git a/airflow/contrib/hooks/aws_firehose_hook.py
b/airflow/contrib/hooks/aws_firehose_hook.py
new file mode 100644
index 000..cf7b2fc
--- /dev/null
+++ b/airflow/contrib/hooks/aws_firehose_hook.py
@@ -0,0 +1,52 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from airflow.contrib.hooks.aws_hook import AwsHook
+
+
+class AwsFirehoseHook(AwsHook):
+"""
+Interact with AWS Kinesis Firehose.
+:param delivery_stream: Name of the delivery stream
+:type delivery_stream: str
+:param region_name: AWS region name (example: us-east-1)
+:type region_name: str
+"""
+
+def __init__(self, delivery_stream, region_name=None, *args, **kwargs):
+self.delivery_stream = delivery_stream
+self.region_name = region_name
+super(AwsFirehoseHook, self).__init__(*args, **kwargs)
+
+def get_conn(self):
+"""
+Returns AwsHook connection object.
+"""
+
+self.conn = self.get_client_type('firehose', self.region_name)
+return self.conn
+
+def put_records(self, records):
+"""
+Write batch records to Kinesis Firehose
+"""
+
+firehose_conn = self.get_conn()
+
+response = firehose_conn.put_record_batch(
+DeliveryStreamName=self.delivery_stream,
+Records=records
+)
+
+return response
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2d588e94/tests/contrib/hooks/test_aws_firehose_hook.py
--
diff --git a/tests/contrib/hooks/test_aws_firehose_hook.py
b/tests/contrib/hooks/test_aws_firehose_hook.py
new file mode 100644
index 000..0a2c809
--- /dev/null
+++ b/tests/contrib/hooks/test_aws_firehose_hook.py
@@ -0,0 +1,70 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import unittest
+import uuid
+
+from airflow.contrib.hooks.aws_firehose_hook import AwsFirehoseHook
+
+try:
+from moto import mock_kinesis
+except ImportError:
+mock_kinesis = None
+
+
+class TestAwsFirehoseHook(unittest.TestCase):
+
+@unittest.skipIf(mock_kinesis is None, 'mock_kinesis package not present')
+@mock_kinesis
+def test_get_conn_returns_a_boto3_connection(self):
+hook = AwsFirehoseHook(aws_conn_id='aws_default',
+ delivery_stream="test_airflow",
region_name="us-east-1")
+self.assertIsNotNone(hook.get_conn())
+
+@unittest.skipIf(mock_kinesis is None, 'mock_kinesis package not present')
+@mock_kinesis
+def test_insert_batch_records_kinesis_firehose(self):
+hook = AwsFirehoseHook(aws_conn_id='aws_default',
+ del
incubator-airflow git commit: closes apache/incubator-airflow#3276 *Messed up PR - hundreds of old commits.*
Repository: incubator-airflow Updated Branches: refs/heads/master e9b74b68a -> e691acc11 closes apache/incubator-airflow#3276 *Messed up PR - hundreds of old commits.* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e691acc1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e691acc1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e691acc1 Branch: refs/heads/master Commit: e691acc115683f1fc6881e6dcdbc9fdedffde667 Parents: e9b74b6 Author: r39132 Authored: Sat Apr 28 14:57:27 2018 -0700 Committer: r39132 Committed: Sat Apr 28 14:57:27 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2827 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master 172b73c15 -> 67c0099e9 closes apache/incubator-airflow#2827 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/67c0099e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/67c0099e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/67c0099e Branch: refs/heads/master Commit: 67c0099e97ae5d679f838d404bbbc52cefdf5201 Parents: 172b73c Author: r39132 Authored: Fri Apr 27 09:46:51 2018 -0700 Committer: r39132 Committed: Fri Apr 27 09:46:51 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#2586 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master b92252158 -> 172b73c15 closes apache/incubator-airflow#2586 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/172b73c1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/172b73c1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/172b73c1 Branch: refs/heads/master Commit: 172b73c155f555c49c340275793c0c0a0b09bc24 Parents: b922521 Author: r39132 Authored: Fri Apr 27 09:42:19 2018 -0700 Committer: r39132 Committed: Fri Apr 27 09:42:19 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#1933 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master 414a08e50 -> b92252158 closes apache/incubator-airflow#1933 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b9225215 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b9225215 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b9225215 Branch: refs/heads/master Commit: b9225215873251c061accd89349b82b99ad5d905 Parents: 414a08e Author: r39132 Authored: Fri Apr 27 09:39:13 2018 -0700 Committer: r39132 Committed: Fri Apr 27 09:39:13 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2378] Add Groupon to list of current users
Repository: incubator-airflow Updated Branches: refs/heads/master ae63246a1 -> 801fe7dbd [AIRFLOW-2378] Add Groupon to list of current users Closes #3267 from stevencasey/add_groupon Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/801fe7db Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/801fe7db Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/801fe7db Branch: refs/heads/master Commit: 801fe7dbdcc74988fe937e013d7f019df87c44e5 Parents: ae63246 Author: steven casey <265229+stevenca...@users.noreply.github.com> Authored: Thu Apr 26 18:47:28 2018 -0700 Committer: r39132 Committed: Thu Apr 26 18:47:28 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/801fe7db/README.md -- diff --git a/README.md b/README.md index 5e5193c..b504cbd 100644 --- a/README.md +++ b/README.md @@ -140,6 +140,7 @@ Currently **officially** using Airflow: 1. [GovTech GDS](https://gds-gov.tech) [[@chrissng](https://github.com/chrissng) & [@datagovsg](https://github.com/datagovsg)] 1. [Grand Rounds](https://www.grandrounds.com/) [[@richddr](https://github.com/richddr), [@timz1290](https://github.com/timz1290), [@wenever](https://github.com/@wenever), & [@runongirlrunon](https://github.com/runongirlrunon)] 1. [Groupalia](http://es.groupalia.com) [[@jesusfcr](https://github.com/jesusfcr)] +1. [Groupon](https://groupon.com) [[@stevencasey](https://github.com/stevencasey)] 1. [Gusto](https://gusto.com) [[@frankhsu](https://github.com/frankhsu)] 1. [Handshake](https://joinhandshake.com/) [[@mhickman](https://github.com/mhickman)] 1. [Handy](http://www.handy.com/careers/73115?gh_jid=73115&gh_src=o5qcxn) [[@marcintustin](https://github.com/marcintustin) / [@mtustin-handy](https://github.com/mtustin-handy)]
incubator-airflow git commit: [AIRFLOW-2382] Fix wrong description for delimiter
Repository: incubator-airflow Updated Branches: refs/heads/master 36193fc74 -> ae63246a1 [AIRFLOW-2382] Fix wrong description for delimiter Fix misleading descriptions for the 'delimiter' parameter in S3ListOperator and S3ToGoogleCloudStorageOperator's docstring. Closes #3270 from sekikn/AIRFLOW-2382 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ae63246a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ae63246a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ae63246a Branch: refs/heads/master Commit: ae63246a1d580d77c366276300139165e4c984de Parents: 36193fc Author: Kengo Seki Authored: Thu Apr 26 18:45:12 2018 -0700 Committer: r39132 Committed: Thu Apr 26 18:45:12 2018 -0700 -- airflow/contrib/operators/s3_list_operator.py | 12 +--- airflow/contrib/operators/s3_to_gcs_operator.py | 6 ++ 2 files changed, 7 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ae63246a/airflow/contrib/operators/s3_list_operator.py -- diff --git a/airflow/contrib/operators/s3_list_operator.py b/airflow/contrib/operators/s3_list_operator.py index 1dcbc60..dbb45fe 100644 --- a/airflow/contrib/operators/s3_list_operator.py +++ b/airflow/contrib/operators/s3_list_operator.py @@ -24,8 +24,7 @@ from airflow.utils.decorators import apply_defaults class S3ListOperator(BaseOperator): """ -List all objects from the bucket with the given string prefix and delimiter -in name. +List all objects from the bucket with the given string prefix in name. This operator returns a python list with the name of objects which can be used by `xcom` in the downstream task. @@ -35,22 +34,21 @@ class S3ListOperator(BaseOperator): :param prefix: Prefix string to filters the objects whose name begin with such prefix :type prefix: string -:param delimiter: The delimiter by which you want to filter the objects. -For e.g to lists the CSV files from in a directory in S3 you would use -delimiter='.csv'. +:param delimiter: the delimiter marks key hierarchy. :type delimiter: string :param aws_conn_id: The connection ID to use when connecting to S3 storage. :type aws_conn_id: string **Example**: -The following operator would list all the CSV files from the S3 +The following operator would list all the files +(excluding subfolders) from the S3 ``customers/2018/04/`` key in the ``data`` bucket. :: s3_file = S3ListOperator( task_id='list_3s_files', bucket='data', prefix='customers/2018/04/', -delimiter='.csv', +delimiter='/', aws_conn_id='aws_customers_conn' ) """ http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ae63246a/airflow/contrib/operators/s3_to_gcs_operator.py -- diff --git a/airflow/contrib/operators/s3_to_gcs_operator.py b/airflow/contrib/operators/s3_to_gcs_operator.py index d105596..5a2004d 100644 --- a/airflow/contrib/operators/s3_to_gcs_operator.py +++ b/airflow/contrib/operators/s3_to_gcs_operator.py @@ -37,12 +37,10 @@ class S3ToGoogleCloudStorageOperator(S3ListOperator): :param prefix: Prefix string which filters objects whose name begin with such prefix. :type prefix: string -:param delimiter: The delimiter by which you want to filter the objects on. -E.g. to list CSV files from a S3 key you would do the following, -`delimiter='.csv'`. +:param delimiter: the delimiter marks key hierarchy. :type delimiter: string :param aws_conn_id: The source S3 connection -:type aws_conn_id: str +:type aws_conn_id: string :param dest_gcs_conn_id: The destination connection ID to use when connecting to Google Cloud Storage. :type dest_gcs_conn_id: string
incubator-airflow git commit: [AIRFLOW-2377] Improve Sendgrid sender support
Repository: incubator-airflow
Updated Branches:
refs/heads/master e44688ed0 -> b0d0d0a04
[AIRFLOW-2377] Improve Sendgrid sender support
Closes #3266 from ms32035/sendgrid_sender
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b0d0d0a0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b0d0d0a0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b0d0d0a0
Branch: refs/heads/master
Commit: b0d0d0a041bd5c4b19f6b97dc0fa289436743272
Parents: e44688e
Author: Marcin Szymanski
Authored: Thu Apr 26 10:25:32 2018 -0700
Committer: r39132
Committed: Thu Apr 26 10:25:32 2018 -0700
--
airflow/contrib/utils/sendgrid.py| 8 +---
tests/contrib/utils/test_sendgrid.py | 31 +++
2 files changed, 32 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b0d0d0a0/airflow/contrib/utils/sendgrid.py
--
diff --git a/airflow/contrib/utils/sendgrid.py
b/airflow/contrib/utils/sendgrid.py
index 206bb93..9055c97 100644
--- a/airflow/contrib/utils/sendgrid.py
+++ b/airflow/contrib/utils/sendgrid.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -51,7 +51,9 @@ def send_email(to, subject, html_content, files=None,
SENDGRID_API_KEY={your-sendgrid-api-key}.
"""
mail = Mail()
-mail.from_email = Email(os.environ.get('SENDGRID_MAIL_FROM'))
+from_email = kwargs.get('from_email') or
os.environ.get('SENDGRID_MAIL_FROM')
+from_name = kwargs.get('from_name') or
os.environ.get('SENDGRID_MAIL_SENDER')
+mail.from_email = Email(from_email, from_name)
mail.subject = subject
# Add the recipient list of to emails.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b0d0d0a0/tests/contrib/utils/test_sendgrid.py
--
diff --git a/tests/contrib/utils/test_sendgrid.py
b/tests/contrib/utils/test_sendgrid.py
index 997bc55..6710076 100644
--- a/tests/contrib/utils/test_sendgrid.py
+++ b/tests/contrib/utils/test_sendgrid.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -50,17 +50,27 @@ class SendEmailSendGridTest(unittest.TestCase):
'subject': 'sendgrid-send-email unit test'}
self.personalization_custom_args = {'arg1': 'val1', 'arg2': 'val2'}
self.categories = ['cat1', 'cat2']
+# extras
self.expected_mail_data_extras = copy.deepcopy(self.expected_mail_data)
self.expected_mail_data_extras['personalizations'][0]['custom_args'] =
\
self.personalization_custom_args
self.expected_mail_data_extras['categories'] = self.categories
+self.expected_mail_data_extras['from'] = \
+{'name': 'Foo', 'email': 'f...@bar.com'}
+# sender
+self.expected_mail_data_sender = copy.deepcopy(self.expected_mail_data)
+self.expected_mail_data_sender['from'] = \
+{'name': 'Foo Bar', 'email': 'f...@foo.bar'}
# Test the right email is constructed.
@mock.patch('os.environ.get')
@mock.patch('airflow.contrib.utils.sendgrid._post_sendgrid_mail')
def test_send_email_sendgrid_correct_email(self, mock_post, mock_get):
-mock_get.return_value = 'f...@bar.com'
+def get_return(var):
+return {'SENDGRID_MAIL_FROM': 'f...@bar.com'}.get(var)
+
+mock_get.side_effect = get_return
send_email(self.to, self.subject, self.html_content, cc=self.cc,
bcc=self.bcc)
mock_post.assert_called_with(self.expected_mail_data)
@@ -68,8 +78,21 @@ class SendEmailSendGridTest(unittest.TestCase):
@mock.patch('os.environ.get')
@mock.patch('airflow.contrib.utils.sendgrid._post_sendgrid_mail')
def test_send_email_sendgrid_correct_email_extras(self, mock_post,
mock_get):
-mock_get.return_value = 'f...@bar.com'
+def get_return(var):
+return {'SENDGRID_MAIL_FROM': 'f...@
incubator-airflow git commit: closes apache/incubator-airflow#3032 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master 2a8bb0e1b -> dde066d00 closes apache/incubator-airflow#3032 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/dde066d0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/dde066d0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/dde066d0 Branch: refs/heads/master Commit: dde066d0042cf444afec3694f3d31f0117d6ff3a Parents: 2a8bb0e Author: r39132 Authored: Wed Apr 25 21:30:25 2018 -0700 Committer: r39132 Committed: Wed Apr 25 21:30:25 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-1835] Update docs: Variable file is json
Repository: incubator-airflow Updated Branches: refs/heads/master ec0d227ef -> 2a8bb0e1b [AIRFLOW-1835] Update docs: Variable file is json Searching through all the documentation I couldn't find anywhere that explained what file format it expected for uploading settings. Closes #2802 from bovard/variable_files_are_json Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2a8bb0e1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2a8bb0e1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2a8bb0e1 Branch: refs/heads/master Commit: 2a8bb0e1b7a15a70519020e69e7369700df5dc37 Parents: ec0d227 Author: Bovard Doerschuk-Tiberi Authored: Wed Apr 25 14:21:22 2018 -0700 Committer: r39132 Committed: Wed Apr 25 14:21:35 2018 -0700 -- docs/concepts.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2a8bb0e1/docs/concepts.rst -- diff --git a/docs/concepts.rst b/docs/concepts.rst index e85238d..89c25fe 100644 --- a/docs/concepts.rst +++ b/docs/concepts.rst @@ -410,7 +410,8 @@ Variables Variables are a generic way to store and retrieve arbitrary content or settings as a simple key value store within Airflow. Variables can be listed, created, updated and deleted from the UI (``Admin -> Variables``), -code or CLI. While your pipeline code definition and most of your constants +code or CLI. In addition, json settings files can be bulk uploaded through +the UI. While your pipeline code definition and most of your constants and variables should be defined in code and stored in source control, it can be useful to have some variables or configuration items accessible and modifiable through the UI.
incubator-airflow git commit: [AIRFLOW-1781] Make search case-insensitive in LDAP group
Repository: incubator-airflow
Updated Branches:
refs/heads/master 8e8c4eb4d -> ec0d227ef
[AIRFLOW-1781] Make search case-insensitive in LDAP group
Closes #2750 from patsak/f/ldap_search
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ec0d227e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ec0d227e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ec0d227e
Branch: refs/heads/master
Commit: ec0d227ef8bc02f63515e7cb011f15d8fcf7730b
Parents: 8e8c4eb
Author: k.privezentsev
Authored: Wed Apr 25 14:07:30 2018 -0700
Committer: r39132
Committed: Wed Apr 25 14:07:53 2018 -0700
--
airflow/contrib/auth/backends/ldap_auth.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ec0d227e/airflow/contrib/auth/backends/ldap_auth.py
--
diff --git a/airflow/contrib/auth/backends/ldap_auth.py
b/airflow/contrib/auth/backends/ldap_auth.py
index c887ffb..1ab5fcd 100644
--- a/airflow/contrib/auth/backends/ldap_auth.py
+++ b/airflow/contrib/auth/backends/ldap_auth.py
@@ -77,12 +77,13 @@ def get_ldap_connection(dn=None, password=None):
def group_contains_user(conn, search_base, group_filter, user_name_attr,
username):
search_filter = '(&({0}))'.format(group_filter)
+
if not conn.search(native(search_base), native(search_filter),
attributes=[native(user_name_attr)]):
log.warning("Unable to find group for %s %s", search_base,
search_filter)
else:
for entry in conn.entries:
-if username in getattr(entry, user_name_attr).values:
+if username.lower() in map(lambda attr: attr.lower(),
getattr(entry, user_name_attr).values):
return True
return False
incubator-airflow git commit: [AIRFLOW-2042] Fix browser menu appearing over the autocomplete menu
Repository: incubator-airflow
Updated Branches:
refs/heads/master 0889770dc -> 8e8c4eb4d
[AIRFLOW-2042] Fix browser menu appearing over the autocomplete menu
Closes #2984 from fox/fix-autocomplete
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8e8c4eb4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8e8c4eb4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8e8c4eb4
Branch: refs/heads/master
Commit: 8e8c4eb4d6a5460de342afa7bd79f6c48dd124cc
Parents: 0889770
Author: Sasa Brankovic
Authored: Wed Apr 25 11:39:26 2018 -0700
Committer: r39132
Committed: Wed Apr 25 11:39:26 2018 -0700
--
airflow/www/templates/airflow/dags.html | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8e8c4eb4/airflow/www/templates/airflow/dags.html
--
diff --git a/airflow/www/templates/airflow/dags.html
b/airflow/www/templates/airflow/dags.html
index eb22708..d22bfb3 100644
--- a/airflow/www/templates/airflow/dags.html
+++ b/airflow/www/templates/airflow/dags.html
@@ -278,9 +278,10 @@
$input.change(function() {
var current = $input.typeahead("getActive");
-
});
+ $input.attr("autocomplete", "off");
+
$('#dags').dataTable({
"iDisplayLength": 500,
"bSort": false,
incubator-airflow git commit: [AIRFLOW-2041] Correct Syntax in python examples
Repository: incubator-airflow
Updated Branches:
refs/heads/master 64d950166 -> fd6f1d1a0
[AIRFLOW-2041] Correct Syntax in python examples
I parsed it with the ol' eyeball compiler. Someone
could flake8 it better, perhaps.
Changes:
- correct `def` syntax on line 50
- use literal dict on line 67
Closes #2479 from 0atman/patch-1
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fd6f1d1a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fd6f1d1a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fd6f1d1a
Branch: refs/heads/master
Commit: fd6f1d1a07a914e23e44b853f81124c9b423d22e
Parents: 64d9501
Author: Tristram Oaten
Authored: Tue Apr 24 23:04:28 2018 -0700
Committer: r39132
Committed: Tue Apr 24 23:04:38 2018 -0700
--
docs/concepts.rst | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fd6f1d1a/docs/concepts.rst
--
diff --git a/docs/concepts.rst b/docs/concepts.rst
index 3f70555..e85238d 100644
--- a/docs/concepts.rst
+++ b/docs/concepts.rst
@@ -47,7 +47,7 @@ scope.
dag_1 = DAG('this_dag_will_be_discovered')
-def my_function()
+def my_function():
dag_2 = DAG('but_this_dag_will_not')
my_function()
@@ -64,9 +64,10 @@ any of its operators. This makes it easy to apply a common
parameter to many ope
.. code:: python
-default_args=dict(
-start_date=datetime(2016, 1, 1),
-owner='Airflow')
+default_args = {
+'start_date': datetime(2016, 1, 1),
+'owner': 'Airflow'
+}
dag = DAG('my_dag', default_args=default_args)
op = DummyOperator(task_id='dummy', dag=dag)
incubator-airflow git commit: closes apache/incubator-airflow#3187 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master 4c02ad76a -> 2c1052d10 closes apache/incubator-airflow#3187 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2c1052d1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2c1052d1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2c1052d1 Branch: refs/heads/master Commit: 2c1052d100c1d5c5bb685d0eb6412dae32416f1b Parents: 4c02ad7 Author: r39132 Authored: Fri Apr 20 09:35:32 2018 -0700 Committer: r39132 Committed: Fri Apr 20 09:35:32 2018 -0700 -- --
incubator-airflow git commit: closes apache/incubator-airflow#3225 *Closed for inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master f1e65c489 -> e6145784e closes apache/incubator-airflow#3225 *Closed for inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e6145784 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e6145784 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e6145784 Branch: refs/heads/master Commit: e6145784e64c6160e302417fed4474fd580e9a8b Parents: f1e65c4 Author: r39132 Authored: Thu Apr 19 18:37:34 2018 -0700 Committer: r39132 Committed: Thu Apr 19 18:37:34 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2347] Add Banco de Formaturas to Readme
Repository: incubator-airflow
Updated Branches:
refs/heads/master c208a5668 -> f1e65c489
[AIRFLOW-2347] Add Banco de Formaturas to Readme
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2347
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Add a company to the README
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason: N/A -- documentation update only
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3242 from
r39132/Add_banco_Formaturas_to_readme
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f1e65c48
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f1e65c48
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f1e65c48
Branch: refs/heads/master
Commit: f1e65c4897535aa9b97f2ce1ae628eddc6a4a6e5
Parents: c208a56
Author: Sid Anand
Authored: Thu Apr 19 18:33:33 2018 -0700
Committer: r39132
Committed: Thu Apr 19 18:33:33 2018 -0700
--
README.md | 1 +
1 file changed, 1 insertion(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f1e65c48/README.md
--
diff --git a/README.md b/README.md
index 943919b..5e5193c 100644
--- a/README.md
+++ b/README.md
@@ -93,6 +93,7 @@ Currently **officially** using Airflow:
1. [Auth0](https://auth0.com) [[@sicarul](https://github.com/sicarul)]
1. [Away](https://awaytravel.com) [[@trunsky](https://github.com/trunsky)]
1. [BalanceHero](http://truebalance.io/)
[[@swalloow](https://github.com/swalloow)]
+1. [Banco de Formaturas](https://www.bancodeformaturas.com.br)
[[@guiligan](https://github.com/guiligan)]
1. [Azri Solutions](http://www.azrisolutions.com/)
[[@userimack](https://github.com/userimack)]
1. [BandwidthX](http://www.bandwidthx.com)
[[@dineshdsharma](https://github.com/dineshdsharma)]
1. [Bellhops](https://github.com/bellhops)
incubator-airflow git commit: [AIRFLOW-2346] Add Investorise as official user of Airflow
Repository: incubator-airflow Updated Branches: refs/heads/master 17d3d1d9d -> c208a5668 [AIRFLOW-2346] Add Investorise as official user of Airflow Closes #3238 from svenvarkel/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c208a566 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c208a566 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c208a566 Branch: refs/heads/master Commit: c208a5668285a4cbd5e1073535f30774e942eac1 Parents: 17d3d1d Author: Sven Varkel Authored: Thu Apr 19 18:20:31 2018 -0700 Committer: r39132 Committed: Thu Apr 19 18:20:37 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c208a566/README.md -- diff --git a/README.md b/README.md index 6911e32..943919b 100644 --- a/README.md +++ b/README.md @@ -155,6 +155,7 @@ Currently **officially** using Airflow: 1. [imgix](https://www.imgix.com/) [[@dclubb](https://github.com/dclubb)] 1. [ING](http://www.ing.com/) 1. [Intercom](http://www.intercom.com/) [[@fox](https://github.com/fox) & [@paulvic](https://github.com/paulvic)] +1. [Investorise](https://investorise.com/) [[@svenvarkel](https://github.com/svenvarkel)] 1. [Jampp](https://github.com/jampp) 1. [JobTeaser](https://www.jobteaser.com) [[@stefani75](https://github.com/stefani75) & [@knil-sama](https://github.com/knil-sama)] 1. [Kalibrr](https://www.kalibrr.com/) [[@charlesverdad](https://github.com/charlesverdad)]
incubator-airflow git commit: [AIRFLOW-2306] Add Bonnier Broadcasting to list of current users
Repository: incubator-airflow Updated Branches: refs/heads/master 44ce9ca71 -> 3beb9bcb5 [AIRFLOW-2306] Add Bonnier Broadcasting to list of current users Closes #3206 from wileeam/add-bbr-to-readme Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3beb9bcb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3beb9bcb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3beb9bcb Branch: refs/heads/master Commit: 3beb9bcb51f66e8a3db56f4712e2a846568bedef Parents: 44ce9ca Author: Guillermo RodrÃguez Cano Authored: Mon Apr 9 14:50:18 2018 -0700 Committer: r39132 Committed: Mon Apr 9 14:50:25 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3beb9bcb/README.md -- diff --git a/README.md b/README.md index 62aa27c..8a5b057 100644 --- a/README.md +++ b/README.md @@ -103,6 +103,7 @@ Currently **officially** using Airflow: 1. [Blue Yonder](http://www.blue-yonder.com) [[@blue-yonder](https://github.com/blue-yonder)] 1. [Boda Telecom Suite - CE](https://github.com/bodastage/bts-ce) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)] 1. [Bodastage Solutions](http://bodastage.com) [[@erssebaggala](https://github.com/erssebaggala), [@bodastage](https://github.com/bodastage)] +1. [Bonnier Broadcasting](http://www.bonnierbroadcasting.com) [[@wileeam](https://github.com/wileeam)] 1. [California Data Collaborative](https://github.com/California-Data-Collaborative) powered by [ARGO Labs](http://www.argolabs.org) 1. [Carbonite](https://www.carbonite.com) [[@ajbosco](https://github.com/ajbosco)] 1. [Celect](http://www.celect.com) [[@superdosh](https://github.com/superdosh) & [@chadcelect](https://github.com/chadcelect)]
incubator-airflow git commit: [AIRFLOW-2287] Update license notices
Repository: incubator-airflow Updated Branches: refs/heads/master f7f1d38ba -> a30f009ae [AIRFLOW-2287] Update license notices Closes #3195 from bolkedebruin/AIRFLOW-2287 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a30f009a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a30f009a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a30f009a Branch: refs/heads/master Commit: a30f009aeb19347524562d3893ab538585183167 Parents: f7f1d38 Author: Bolke de Bruin Authored: Mon Apr 9 00:32:02 2018 -0700 Committer: r39132 Committed: Mon Apr 9 00:32:09 2018 -0700 -- LICENSE | 24 + dev/airflow-license | 71 ++ licenses/LICENSE-bootstrap-toggle.txt | 21 licenses/LICENSE-bootstrap.txt| 25 - licenses/LICENSE-jquery.txt | 33 licenses/LICENSE-jsclockplugin.txt| 7 --- licenses/LICENSE-normalize.txt| 21 licenses/LICENSE-parallel-coordinates.txt | 26 ++ licenses/LICENSE-typeahead.txt| 13 - scripts/ci/check-license.sh | 2 +- 10 files changed, 155 insertions(+), 88 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a30f009a/LICENSE -- diff --git a/LICENSE b/LICENSE index d468094..540289b 100644 --- a/LICENSE +++ b/LICENSE @@ -214,28 +214,34 @@ MIT licenses The following components are provided under the MIT License. See project link for details. -The text of each license is also included at licenses/LICENSE-[project]-[version].txt. +The text of each license is also included at licenses/LICENSE-[project].txt. (MIT License) jquery (https://jquery.org/license/) (MIT License) dagre-d3 (https://github.com/cpettitt/dagre-d3) -(MIT License) bootstrap (https://github.com/twbs/bootstrap/blob/v3-dev/LICENSE) +(MIT License) bootstrap (https://github.com/twbs/bootstrap/) (MIT License) d3-tip (https://github.com/Caged/d3-tip) (MIT License) dataTables (https://datatables.net) -(MIT License) Clock Plugin (https://github.com/Lwangaman/jQuery-Clock-Plugin) (MIT License) WebGL-2D (https://github.com/gameclosure/webgl-2d) -(MIT License) Underscore.js (http://underscorejs.org) +(MIT License) Underscorejs (http://underscorejs.org) (MIT License) Bootstrap Toggle (http://www.bootstraptoggle.com) -(MIT License) normalize.css (http://git.io/normalize) +(MIT License) normalize.css (http://necolas.github.io/normalize.css/) BSD 2-Clause licenses The following components are provided under the BSD 2-Clause license. See file headers and project links for details. -The text of each license is also included at licenses/LICENSE-[project]-[version].txt. +The text of each license is also included at licenses/LICENSE-[project].txt. (BSD 2 License) flask-kerberos (https://github.com/mkomitee/flask-kerberos) -(BSD 2 License) Ace (https://github.com/ajaxorg/ace) -(BSD 2 License) d3js (https://d3js.org) -(BSD 3 License) parallel.js (https://parallel.js.org/) + +BSD 3-Clause licenses + +The following components are provided under the BSD 2-Clause license. +See file headers and project links for details. +The text of each license is also included at licenses/LICENSE-[project].txt. + +(BSD 3 License) Ace (https://github.com/ajaxorg/ace) +(BSD 3 License) d3js (https://d3js.org) +(BSD 3 License) parallel-coordinates (http://syntagmatic.github.com/parallel-coordinates/) http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a30f009a/dev/airflow-license -- diff --git a/dev/airflow-license b/dev/airflow-license new file mode 100755 index 000..b7c7ad6 --- /dev/null +++ b/dev/airflow-license @@ -0,0 +1,71 @@ +#!/usr/bin/env python + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License
incubator-airflow git commit: Closes apache/incubator-airflow#3192 *Already Merged*
Repository: incubator-airflow Updated Branches: refs/heads/master 347acbb03 -> f7f1d38ba Closes apache/incubator-airflow#3192 *Already Merged* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f7f1d38b Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f7f1d38b Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f7f1d38b Branch: refs/heads/master Commit: f7f1d38badf836814fa183e0925d0584bfcaf6fc Parents: 347acbb Author: r39132 Authored: Sun Apr 8 13:03:27 2018 -0700 Committer: r39132 Committed: Sun Apr 8 13:03:27 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2296] Add Cinimex DataLab to Readme
Repository: incubator-airflow Updated Branches: refs/heads/master 9b1bfb588 -> 347acbb03 [AIRFLOW-2296] Add Cinimex DataLab to Readme Closes #3198 from r39132/add_cinemex_to_readme Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/347acbb0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/347acbb0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/347acbb0 Branch: refs/heads/master Commit: 347acbb03448bad3f577fb38d49f35ddc449c4e0 Parents: 9b1bfb5 Author: r39132 Authored: Sun Apr 8 12:50:46 2018 -0700 Committer: r39132 Committed: Sun Apr 8 12:51:01 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/347acbb0/README.md -- diff --git a/README.md b/README.md index 09b80c6..62aa27c 100644 --- a/README.md +++ b/README.md @@ -109,6 +109,7 @@ Currently **officially** using Airflow: 1. [Change.org](https://www.change.org) [[@change](https://github.com/change), [@vijaykramesh](https://github.com/vijaykramesh)] 1. [Checkr](https://checkr.com) [[@tongboh](https://github.com/tongboh)] 1. [Children's Hospital of Philadelphia Division of Genomic Diagnostics](http://www.chop.edu/centers-programs/division-genomic-diagnostics) [[@genomics-geek]](https://github.com/genomics-geek/) +1. [Cinimex DataLab](http://cinimex.ru) [[@kdubovikov](https://github.com/kdubovikov)] 1. [City of San Diego](http://sandiego.gov) [[@MrMaksimize](https://github.com/mrmaksimize), [@andrell81](https://github.com/andrell81) & [@arnaudvedy](https://github.com/arnaudvedy)] 1. [Clairvoyant](https://clairvoyantsoft.com) [@shekharv](https://github.com/shekharv) 1. [Clover Health](https://www.cloverhealth.com) [[@gwax](https://github.com/gwax) & [@vansivallab](https://github.com/vansivallab)]
incubator-airflow git commit: *Kick mirroring again*
Repository: incubator-airflow Updated Branches: refs/heads/master 04262eb9b -> a15bd8048 *Kick mirroring again* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a15bd804 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a15bd804 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a15bd804 Branch: refs/heads/master Commit: a15bd804815aac74a4ca4b8cf6985dd7455a7889 Parents: 04262eb Author: Siddharth Anand Authored: Thu Apr 5 22:18:36 2018 -0700 Committer: Siddharth Anand Committed: Thu Apr 5 22:18:36 2018 -0700 -- --
incubator-airflow git commit: *Kick mirroring*
Repository: incubator-airflow Updated Branches: refs/heads/master dcfadcdf4 -> 04262eb9b *Kick mirroring* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/04262eb9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/04262eb9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/04262eb9 Branch: refs/heads/master Commit: 04262eb9b6e6bbfb0cf0d5872e34b4d020e378c1 Parents: dcfadcd Author: Siddharth Anand Authored: Thu Apr 5 19:07:14 2018 -0700 Committer: Siddharth Anand Committed: Thu Apr 5 19:07:14 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2286] Add tokopedia to the readme
Repository: incubator-airflow Updated Branches: refs/heads/master 9804f48d4 -> dcfadcdf4 [AIRFLOW-2286] Add tokopedia to the readme Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/dcfadcdf Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/dcfadcdf Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/dcfadcdf Branch: refs/heads/master Commit: dcfadcdf4d3b159592569245d85b8b51de38beea Parents: 9804f48 Author: Siddharth Anand Authored: Thu Apr 5 19:01:09 2018 -0700 Committer: Siddharth Anand Committed: Thu Apr 5 19:01:09 2018 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/dcfadcdf/README.md -- diff --git a/README.md b/README.md index 5c69c4c..0c0baa8 100644 --- a/README.md +++ b/README.md @@ -212,6 +212,7 @@ Currently **officially** using Airflow: 1. [Thumbtack](https://www.thumbtack.com/) [[@natekupp](https://github.com/natekupp)] 1. [Tictail](https://tictail.com/) 1. [Tile](https://tile.com/) [[@ranjanmanish](https://github.com/ranjanmanish)] +1. [Tokopedia](https://www.tokopedia.com/) [@topedmaria](https://github.com/topedmaria) 1. [T2 Systems](http://t2systems.com) [[@unclaimedpants](https://github.com/unclaimedpants)] 1. [Ubisoft](https://www.ubisoft.com/) [[@Walkoss](https://github.com/Walkoss)] 1. [United Airlines](https://www.united.com/) [[@ilopezfr](https://github.com/ilopezfr)]
incubator-airflow git commit: Closes apache/incubator-airflow#2953 *No movement from submitter* [Forced Update!]
Repository: incubator-airflow Updated Branches: refs/heads/master b918a4976 -> 9804f48d4 (forced update) Closes apache/incubator-airflow#2953 *No movement from submitter* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9804f48d Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9804f48d Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9804f48d Branch: refs/heads/master Commit: 9804f48d46a39a4ded443cdf6fa805d14312dacf Parents: 86f30de Author: Siddharth Anand Authored: Thu Apr 5 18:45:52 2018 -0700 Committer: Siddharth Anand Committed: Thu Apr 5 18:45:52 2018 -0700 -- --
[2/3] incubator-airflow git commit: temp_patch
temp_patch
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8231c19a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8231c19a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8231c19a
Branch: refs/heads/master
Commit: 8231c19a412460b7731d1b6ec2bb8fd22128d6b0
Parents: 86f30de
Author: Siddharth Anand
Authored: Mon Mar 5 18:17:18 2018 +
Committer: Siddharth Anand
Committed: Thu Apr 5 18:40:31 2018 -0700
--
dev/airflow-pr | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8231c19a/dev/airflow-pr
--
diff --git a/dev/airflow-pr b/dev/airflow-pr
index 65243dd..d485c12 100755
--- a/dev/airflow-pr
+++ b/dev/airflow-pr
@@ -825,12 +825,15 @@ def main(pr_num, local=False):
os.chdir(AIRFLOW_GIT_LOCATION)
original_head = get_current_ref()
-
+
+print('GOT HERE 1')
branches = get_json("%s/branches" % GITHUB_API_BASE)
+print('GOT HERE 2')
branch_names = filter(
lambda x: x.startswith("branch-"), [x['name'] for x in branches])
# Assumes branch names can be sorted lexicographically
latest_branch = sorted(branch_names, reverse=True)
+print('GOT HERE 3')
if latest_branch:
latest_branch = latest_branch[0]
else:
[1/3] incubator-airflow git commit: temp_patch_2
Repository: incubator-airflow
Updated Branches:
refs/heads/master 86f30debd -> b918a4976
temp_patch_2
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/1d16859f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/1d16859f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/1d16859f
Branch: refs/heads/master
Commit: 1d16859f985b94024aa76017268bd7877ae1fccf
Parents: 8231c19
Author: Siddharth Anand
Authored: Mon Mar 5 18:20:38 2018 +
Committer: Siddharth Anand
Committed: Thu Apr 5 18:40:31 2018 -0700
--
dev/airflow-pr | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1d16859f/dev/airflow-pr
--
diff --git a/dev/airflow-pr b/dev/airflow-pr
index d485c12..17c6354 100755
--- a/dev/airflow-pr
+++ b/dev/airflow-pr
@@ -107,12 +107,14 @@ def reflow(text, width=80):
def get_json(url):
try:
+print('GOT HERE 11')
request = urllib.Request(url)
if GITHUB_OAUTH_KEY:
request.add_header('Authorization', 'token %s' % GITHUB_OAUTH_KEY)
-
+print('GOT HERE 12')
# decode response for Py3 compatibility
response = urllib.urlopen(request).read().decode('utf-8')
+print('GOT HERE 13')
return json.loads(response)
except urllib.HTTPError as e:
if (
[3/3] incubator-airflow git commit: Closes apache/incubator-airflow#2953 *No movement from submitter*
Closes apache/incubator-airflow#2953 *No movement from submitter* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b918a497 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b918a497 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b918a497 Branch: refs/heads/master Commit: b918a4976a641a3c944ef4ad4323fe25bd219ea0 Parents: 1d16859 Author: Siddharth Anand Authored: Thu Apr 5 18:41:14 2018 -0700 Committer: Siddharth Anand Committed: Thu Apr 5 18:41:14 2018 -0700 -- --
incubator-airflow git commit: [AIRFLOW-2134] Add Alan to the list of companies that use Airflow
Repository: incubator-airflow Updated Branches: refs/heads/master 7b0493a42 -> f30253cb3 [AIRFLOW-2134] Add Alan to the list of companies that use Airflow Closes #3041 from Charles-Go/patch-1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f30253cb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f30253cb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f30253cb Branch: refs/heads/master Commit: f30253cb3ee6996185891c9765b5eda86bf0a368 Parents: 7b0493a Author: Charles Gorintin Authored: Wed Feb 21 13:09:48 2018 -0800 Committer: Siddharth Anand Committed: Wed Feb 21 13:09:52 2018 -0800 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f30253cb/README.md -- diff --git a/README.md b/README.md index 6deb405..7393ea0 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,7 @@ Currently **officially** using Airflow: 1. [Airbnb](http://airbnb.io/) [[@mistercrunch](https://github.com/mistercrunch), [@artwr](https://github.com/artwr)] 1. [Agari](https://github.com/agaridata) [[@r39132](https://github.com/r39132)] +1. [Alan](https://alan.eu) [[@charles-go](https://github.com/charles-go)] 1. [allegro.pl](http://allegro.tech/) [[@kretes](https://github.com/kretes)] 1. [AltX](https://www.getaltx.com/about) [[@pedromduarte](https://github.com/pedromduarte)] 1. [Apigee](https://apigee.com) [[@btallman](https://github.com/btallman)]
incubator-airflow git commit: [AIRFLOW-2133] Remove references to GitHub issues in CONTRIBUTING
Repository: incubator-airflow Updated Branches: refs/heads/master 6c93460b9 -> 7b0493a42 [AIRFLOW-2133] Remove references to GitHub issues in CONTRIBUTING Since GitHub issues are now disabled for the project, this guide shouldn't reference them⦠Closes #3060 from reidab/patch-1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7b0493a4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7b0493a4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7b0493a4 Branch: refs/heads/master Commit: 7b0493a42f2b368664b7d34f303fe71c962c0312 Parents: 6c93460 Author: Reid Beels Authored: Wed Feb 21 13:00:54 2018 -0800 Committer: Siddharth Anand Committed: Wed Feb 21 13:01:04 2018 -0800 -- CONTRIBUTING.md | 6 ++ 1 file changed, 2 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7b0493a4/CONTRIBUTING.md -- diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6ac8c43..44ce502 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -29,7 +29,6 @@ Report bugs through [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) Please report relevant information and preferably code that exhibits the problem. - ### Fix Bugs Look through the Jira issues for bugs. Anything is open to whoever wants @@ -37,8 +36,7 @@ to implement it. ### Implement Features -Look through the GitHub issues for features. Anything tagged with -"feature" is open to whoever wants to implement it. +Look through the [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) for features. Any unassigned "Improvement" issue is open to whoever wants to implement it. We've created the operators, hooks, macros and executors we needed, but we made sure that this part of Airflow is extensible. New operators, @@ -53,7 +51,7 @@ articles. ### Submit Feedback -The best way to send feedback is to file an issue on Github. +The best way to send feedback is to open an issue on [Apache Jira](https://issues.apache.org/jira/browse/AIRFLOW) If you are proposing a feature:
incubator-airflow git commit: [AIRFLOW-2126] Add Bluecore to active users
Repository: incubator-airflow Updated Branches: refs/heads/master 3fe06e9ff -> fc26cade8 [AIRFLOW-2126] Add Bluecore to active users Closes #3057 from JLDLaughlin/add_bluecore Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fc26cade Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fc26cade Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fc26cade Branch: refs/heads/master Commit: fc26cade87e181f162ecc8391ae16dccbe6f29c4 Parents: 3fe06e9 Author: Jessica Laughlin Authored: Tue Feb 20 09:07:11 2018 -0800 Committer: Siddharth Anand Committed: Tue Feb 20 09:07:18 2018 -0800 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fc26cade/README.md -- diff --git a/README.md b/README.md index 602b5f0..6deb405 100644 --- a/README.md +++ b/README.md @@ -98,6 +98,7 @@ Currently **officially** using Airflow: 1. [BlaBlaCar](https://www.blablacar.com) [[@puckel](https://github.com/puckel) & [@wmorin](https://github.com/wmorin)] 1. [Bloc](https://www.bloc.io) [[@dpaola2](https://github.com/dpaola2)] 1. [BlueApron](https://www.blueapron.com) [[@jasonjho](https://github.com/jasonjho) & [@matthewdavidhauser](https://github.com/matthewdavidhauser)] +1. [Bluecore](https://www.bluecore.com) [[@JLDLaughlin](https://github.com/JLDLaughlin)] 1. [Blue Yonder](http://www.blue-yonder.com) [[@blue-yonder](https://github.com/blue-yonder)] 1. [California Data Collaborative](https://github.com/California-Data-Collaborative) powered by [ARGO Labs](http://www.argolabs.org) 1. [Carbonite](https://www.carbonite.com) [[@ajbosco](https://github.com/ajbosco)]
incubator-airflow git commit: [AIRFLOW-2057] Add Overstock to list of companies
Repository: incubator-airflow Updated Branches: refs/heads/master 6d88744be -> ba0b1978d [AIRFLOW-2057] Add Overstock to list of companies Closes #3001 from mhousley/add-overstock-to-list Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ba0b1978 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ba0b1978 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ba0b1978 Branch: refs/heads/master Commit: ba0b1978d3c30d4f582115d18953969ecf6ba1ee Parents: 6d88744 Author: Matthew Housley Authored: Thu Feb 1 17:17:18 2018 -0800 Committer: Siddharth Anand Committed: Thu Feb 1 17:17:24 2018 -0800 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ba0b1978/README.md -- diff --git a/README.md b/README.md index 3584d34..c338c17 100644 --- a/README.md +++ b/README.md @@ -169,6 +169,7 @@ Currently **officially** using Airflow: 1. [OfferUp](https://offerupnow.com) 1. [OneFineStay](https://www.onefinestay.com) [[@slangwald](https://github.com/slangwald)] 1. [Open Knowledge International](https://okfn.org) [@vitorbaptista](https://github.com/vitorbaptista) +1. [Overstock](https://www.github.com/overstock) [[@mhousley](https://github.com/mhousley) & [@mct0006](https://github.com/mct0006)] 1. [Pandora Media](https://www.pandora.com/) [[@Acehaidrey](https://github.com/Acehaidrey)] 1. [PAYMILL](https://www.paymill.com/) [[@paymill](https://github.com/paymill) & [@matthiashuschle](https://github.com/matthiashuschle)] 1. [PayPal](https://www.paypal.com/) [[@r39132](https://github.com/r39132) & [@jhsenjaliya](https://github.com/jhsenjaliya)]
incubator-airflow git commit: Kick mirroring
Repository: incubator-airflow Updated Branches: refs/heads/master 2b6d11227 -> cbc02da48 Kick mirroring Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/cbc02da4 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/cbc02da4 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/cbc02da4 Branch: refs/heads/master Commit: cbc02da48f7d5c3e01669a7c9b881feda4eaeda0 Parents: 2b6d112 Author: Siddharth Anand Authored: Wed Jan 24 15:46:05 2018 -0800 Committer: Siddharth Anand Committed: Wed Jan 24 15:46:05 2018 -0800 -- --
incubator-airflow git commit: [AIRFLOW-2028] Add JobTeaser to official users list
Repository: incubator-airflow Updated Branches: refs/heads/master f66282277 -> 2b6d11227 [AIRFLOW-2028] Add JobTeaser to official users list Closes #2966 from stefani75/master Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2b6d1122 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2b6d1122 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2b6d1122 Branch: refs/heads/master Commit: 2b6d112271f0b97333860212cc9b79f8d59c2c6e Parents: f662822 Author: Stéphanie Baltus Authored: Wed Jan 24 15:39:13 2018 -0800 Committer: Siddharth Anand Committed: Wed Jan 24 15:39:24 2018 -0800 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2b6d1122/README.md -- diff --git a/README.md b/README.md index 7fb6212..6659bf1 100644 --- a/README.md +++ b/README.md @@ -143,6 +143,7 @@ Currently **officially** using Airflow: 1. [imgix](https://www.imgix.com/) [[@dclubb](https://github.com/dclubb)] 1. [ING](http://www.ing.com/) 1. [Jampp](https://github.com/jampp) +1. [JobTeaser](https://www.jobteaser.com) [[@stefani75](https://github.com/stefani75) & [@knil-sama](https://github.com/knil-sama)] 1. [Kiwi.com](https://kiwi.com/) [[@underyx](https://github.com/underyx)] 1. [Kogan.com](https://github.com/kogan) [[@geeknam](https://github.com/geeknam)] 1. [Lemann Foundation](http://fundacaolemann.org.br) [[@fernandosjp](https://github.com/fernandosjp)]
[2/2] incubator-airflow git commit: Closes apache/incubator-airflow#1839 *No movement from submitter*
Closes apache/incubator-airflow#1839 *No movement from submitter* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f6628227 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f6628227 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f6628227 Branch: refs/heads/master Commit: f662822772e63d0ad64b6b7eda2bf46ba8abcd61 Parents: b780f39 Author: Siddharth Anand Authored: Wed Jan 24 15:16:45 2018 -0800 Committer: Siddharth Anand Committed: Wed Jan 24 15:16:45 2018 -0800 -- --
[1/2] incubator-airflow git commit: Closes apache/incubator-airflow#730 *No movement from submitter*
Repository: incubator-airflow Updated Branches: refs/heads/master 0990ba8c0 -> f66282277 Closes apache/incubator-airflow#730 *No movement from submitter* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b780f39e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b780f39e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b780f39e Branch: refs/heads/master Commit: b780f39ea48826f65da745769832365b1dc6d922 Parents: 0990ba8 Author: Siddharth Anand Authored: Wed Jan 24 15:16:07 2018 -0800 Committer: Siddharth Anand Committed: Wed Jan 24 15:16:07 2018 -0800 -- --
incubator-airflow git commit: Closes apache/incubator-airflow#2873 *Not merging due to inactivity*
Repository: incubator-airflow Updated Branches: refs/heads/master c69080576 -> bc72231b1 Closes apache/incubator-airflow#2873 *Not merging due to inactivity* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/bc72231b Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/bc72231b Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/bc72231b Branch: refs/heads/master Commit: bc72231b130fcb3f953ed71cbcbdafff5566e556 Parents: c690805 Author: Siddharth Anand Authored: Wed Jan 17 17:37:22 2018 -0800 Committer: Siddharth Anand Committed: Wed Jan 17 17:37:22 2018 -0800 -- --
incubator-airflow git commit: Closes apache/incubator-airflow#2919 *Not merging*
Repository: incubator-airflow Updated Branches: refs/heads/master 59e359819 -> c69080576 Closes apache/incubator-airflow#2919 *Not merging* Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c6908057 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c6908057 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c6908057 Branch: refs/heads/master Commit: c69080576b0b34dd5c833fa41787f77e57a342a9 Parents: 59e3598 Author: Siddharth Anand Authored: Wed Jan 17 17:35:13 2018 -0800 Committer: Siddharth Anand Committed: Wed Jan 17 17:35:13 2018 -0800 -- --
