incubator-airflow git commit: [AIRFLOW-2734] Resolve setuptools normalized_version warning
Repository: incubator-airflow Updated Branches: refs/heads/master 1656209cb -> d704dc357 [AIRFLOW-2734] Resolve setuptools normalized_version warning Closes #3588 from tedmiston/normalized-version Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d704dc35 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d704dc35 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d704dc35 Branch: refs/heads/master Commit: d704dc3572d5c3a29c948b347b13c9f243ec1028 Parents: 1656209 Author: Taylor D. Edmiston Authored: Mon Jul 23 09:41:34 2018 -0700 Committer: Maxime Beauchemin Committed: Mon Jul 23 09:41:34 2018 -0700 -- airflow/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/d704dc35/airflow/version.py -- diff --git a/airflow/version.py b/airflow/version.py index d11d766..6b892ac 100644 --- a/airflow/version.py +++ b/airflow/version.py @@ -18,4 +18,4 @@ # under the License. # -version = '2.0.0dev0+incubating' +version = '2.0.0.dev0+incubating'
[2/2] incubator-airflow git commit: [AIRFLOW-2267] Airflow DAG level access
[AIRFLOW-2267] Airflow DAG level access
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2267
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Provide DAG level access for airflow. The detail
design could be found at https://docs.google.com/d
ocument/d/1qs26lE9kAuCY0Qa0ga-80EQ7d7m4s-590lhjtMB
jmxw/edit#
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
Unit tests are added.
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3197 from feng-tao/airflow-2267
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f3f2eb32
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f3f2eb32
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f3f2eb32
Branch: refs/heads/master
Commit: f3f2eb323f29d2210877ce9699e2dd0bdd8b5259
Parents: 17ddbcf
Author: Tao feng
Authored: Mon Jul 16 13:13:42 2018 -0700
Committer: Maxime Beauchemin
Committed: Mon Jul 16 13:13:42 2018 -0700
--
UPDATING.md | 8 +
airflow/bin/cli.py | 18 +
airflow/settings.py | 5 +-
airflow/www/views.py| 2 +-
airflow/www_rbac/app.py | 25 +-
airflow/www_rbac/decorators.py | 34 +-
airflow/www_rbac/security.py| 393 ++--
airflow/www_rbac/views.py | 220 +--
tests/core.py | 18 +-
tests/jobs.py | 6 +-
tests/models.py | 2 +-
tests/utils/test_db.py | 3 +
tests/www/test_views.py | 2 -
.../www_rbac/api/experimental/test_endpoints.py | 84 +--
tests/www_rbac/test_security.py | 104 +++-
tests/www_rbac/test_views.py| 592 ++-
16 files changed, 1285 insertions(+), 231 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3f2eb32/UPDATING.md
--
diff --git a/UPDATING.md b/UPDATING.md
index 74a0b1b..2ca421d 100644
--- a/UPDATING.md
+++ b/UPDATING.md
@@ -5,6 +5,14 @@ assists users migrating to a new version.
## Airflow Master
+### DAG level Access Control for new RBAC UI
+
+Extend and enhance new Airflow RBAC UI to support DAG level ACL. Each dag now
has two permissions(one for write, one for read) associated('can_dag_edit',
'can_dag_read').
+The admin will create new role, associate the dag permission with the target
dag and assign that role to users. That user can only access / view the certain
dags on the UI
+that he has permissions on. If a new role wants to access all the dags, the
admin could associate dag permissions on an artificial view(``all_dags``) with
that role.
+
+We also provide a new cli command(``sync_perm``) to allow admin to auto sync
permissions.
+
### Setting UTF-8 as default mime_charset in email utils
### Add a configuration variable(default_dag_run_display_number) to control
numbers of dag run for display
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3f2eb32/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 1ecc519..dc31d06 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -1282,6 +1282,9 @@ def create_user(args):
if password != password_confirmation:
raise SystemExit('Passwords did not match!')
+if appbuilder.sm.find_user(args.username):
+print('{} already exist in the db'.format(args.username))
+
[1/2] incubator-airflow git commit: [AIRFLOW-2267] Airflow DAG level access
Repository: incubator-airflow
Updated Branches:
refs/heads/master 17ddbcff0 -> f3f2eb323
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3f2eb32/tests/www_rbac/test_views.py
--
diff --git a/tests/www_rbac/test_views.py b/tests/www_rbac/test_views.py
index 0c25972..b4a2f02 100644
--- a/tests/www_rbac/test_views.py
+++ b/tests/www_rbac/test_views.py
@@ -29,12 +29,11 @@ import unittest
import urllib
from flask._compat import PY2
-from flask_appbuilder.security.sqla.models import User as ab_user
from urllib.parse import quote_plus
from werkzeug.test import Client
from airflow import configuration as conf
-from airflow import models
+from airflow import models, settings
from airflow.config_templates.airflow_local_settings import
DEFAULT_LOGGING_CONFIG
from airflow.models import DAG, DagRun, TaskInstance
from airflow.operators.dummy_operator import DummyOperator
@@ -48,17 +47,17 @@ from airflow.www_rbac import app as application
class TestBase(unittest.TestCase):
def setUp(self):
conf.load_test_config()
-self.app, self.appbuilder = application.create_app(testing=True)
+self.app, self.appbuilder = application.create_app(session=Session,
testing=True)
self.app.config['WTF_CSRF_ENABLED'] = False
self.client = self.app.test_client()
-self.session = Session()
+settings.configure_orm()
+self.session = Session
self.login()
def login(self):
-sm_session = self.appbuilder.sm.get_session()
-self.user = sm_session.query(ab_user).first()
-if not self.user:
-role_admin = self.appbuilder.sm.find_role('Admin')
+role_admin = self.appbuilder.sm.find_role('Admin')
+tester = self.appbuilder.sm.find_user(username='test')
+if not tester:
self.appbuilder.sm.add_user(
username='test',
first_name='test',
@@ -88,6 +87,15 @@ class TestBase(unittest.TestCase):
else:
self.assertIn(text, resp_html)
+def check_content_not_in_response(self, text, resp, resp_code=200):
+resp_html = resp.data.decode('utf-8')
+self.assertEqual(resp_code, resp.status_code)
+if isinstance(text, list):
+for kw in text:
+self.assertNotIn(kw, resp_html)
+else:
+self.assertNotIn(text, resp_html)
+
def percent_encode(self, obj):
if PY2:
return urllib.quote_plus(str(obj))
@@ -137,10 +145,6 @@ class TestVariableModelView(TestBase):
resp = self.client.post('/variable/add',
data=self.variable,
follow_redirects=True)
-self.assertEqual(resp.status_code, 200)
-v = self.session.query(models.Variable).first()
-self.assertEqual(v.key, 'test_key')
-self.assertEqual(v.val, 'text_val')
# update the variable with a wrong value, given that is encrypted
Var = models.Variable
@@ -248,6 +252,8 @@ class TestAirflowBaseViews(TestBase):
def setUp(self):
super(TestAirflowBaseViews, self).setUp()
+self.logout()
+self.login()
self.cleanup_dagruns()
self.prepare_dagruns()
@@ -292,7 +298,7 @@ class TestAirflowBaseViews(TestBase):
self.check_content_in_response('DAGs', resp)
def test_health(self):
-resp = self.client.get('health')
+resp = self.client.get('health', follow_redirects=True)
self.check_content_in_response('The server is healthy!', resp)
def test_home(self):
@@ -361,7 +367,7 @@ class TestAirflowBaseViews(TestBase):
self.check_content_in_response('example_bash_operator', resp)
def test_landing_times(self):
-url = 'landing_times?days=30&dag_id=test_example_bash_operator'
+url = 'landing_times?days=30&dag_id=example_bash_operator'
resp = self.client.get(url, follow_redirects=True)
self.check_content_in_response('example_bash_operator', resp)
@@ -415,6 +421,8 @@ class TestAirflowBaseViews(TestBase):
class TestConfigurationView(TestBase):
def test_configuration(self):
+self.logout()
+self.login()
resp = self.client.get('configuration', follow_redirects=True)
self.check_content_in_response(
['Airflow Configuration', 'Running Configuration'], resp)
@@ -437,6 +445,7 @@ class TestLogView(TestBase):
current_dir = os.path.dirname(os.path.abspath(__file__))
logging_config['handlers']['task']['base_log_folder'] =
os.path.normpath(
os.path.join(current_dir, 'test_logs'))
+
logging_config['handlers']['task']['filename_template'] = \
'{{ ti.dag_id }}/{{ ti.task_id }}/' \
'{{ ts | replace(":", ".") }}/{{ try_number }}.log'
@@ -450,11 +459,12 @@ class TestLogView(TestBase):
sys.path.ap
incubator-airflow git commit: [AIRFLOW-2652] implement / enhance baseOperator deepcopy
Repository: incubator-airflow
Updated Branches:
refs/heads/master 55b56a433 -> 07db7a3d7
[AIRFLOW-2652] implement / enhance baseOperator deepcopy
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2652
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
When running ``airflow backfill`` on
pythonOperator, it will do / trigger a deepcopy of
the task_instance. If some objects can't be
deepcopy in certain python version(e.g Protobuf in
python 2.7) , an exception will be thrown. We
should just do a shallow copy instead of deep copy
for the object.
The pr here is to copy the ``_deepcopy__`` method
in BaseOperator, but skip doing deepcopy for
`op_kwargs` and `python_callable`.
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
I can't think of a good way to test. We encounter
this in our production.
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3528 from feng-tao/airflow-2652
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/07db7a3d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/07db7a3d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/07db7a3d
Branch: refs/heads/master
Commit: 07db7a3d711500c421f73e57002e87b662a2ab0f
Parents: 55b56a4
Author: Tao feng
Authored: Thu Jun 21 16:28:30 2018 -0700
Committer: Maxime Beauchemin
Committed: Thu Jun 21 16:28:30 2018 -0700
--
airflow/models.py| 23 ++-
airflow/operators/python_operator.py | 4
tests/operators/python_operator.py | 21 +++--
3 files changed, 37 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/07db7a3d/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 8f83586..381e9d3 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2317,6 +2317,15 @@ class BaseOperator(LoggingMixin):
ui_color = '#fff'
ui_fgcolor = '#000'
+# base list which includes all the attrs that don't need deep copy.
+_base_operator_shallow_copy_attrs = ('user_defined_macros',
+ 'user_defined_filters',
+ 'params',
+ '_log',)
+
+# each operator should override this class attr for shallow copy attrs.
+shallow_copy_attrs = ()
+
@apply_defaults
def __init__(
self,
@@ -2683,17 +2692,13 @@ class BaseOperator(LoggingMixin):
result = cls.__new__(cls)
memo[id(self)] = result
+shallow_copy = cls.shallow_copy_attrs +
cls._base_operator_shallow_copy_attrs
+
for k, v in list(self.__dict__.items()):
-if k not in ('user_defined_macros', 'user_defined_filters',
- 'params', '_log'):
+if k not in shallow_copy:
setattr(result, k, copy.deepcopy(v, memo))
-result.params = self.params
-if hasattr(self, 'user_defined_macros'):
-result.user_defined_macros = self.user_defined_macros
-if hasattr(self, 'user_defined_filters'):
-result.user_defined_filters = self.user_defined_filters
-if hasattr(self, '_log'):
-result._log = self._log
+else:
+setattr(result, k, copy.copy(v))
return result
def __getstate__(self):
http://git-wip-us.apache.org/repos/a
incubator-airflow git commit: [AIRFLOW-2566] Change backfill to rerun failed tasks
Repository: incubator-airflow
Updated Branches:
refs/heads/master 9c67ee842 -> e9f3fdc52
[AIRFLOW-2566] Change backfill to rerun failed tasks
Closes #3464 from feng-tao/airflow-2566
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e9f3fdc5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e9f3fdc5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e9f3fdc5
Branch: refs/heads/master
Commit: e9f3fdc52cb53f3ac3e9721e5128d17d1c5c418c
Parents: 9c67ee8
Author: Tao feng
Authored: Fri Jun 8 08:28:22 2018 -0700
Committer: Maxime Beauchemin
Committed: Fri Jun 8 08:28:22 2018 -0700
--
airflow/bin/cli.py | 27 +--
airflow/jobs.py | 83 +++--
airflow/models.py | 4 +-
airflow/settings.py | 2 +-
tests/jobs.py | 116 +++
5 files changed, 202 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e9f3fdc5/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 2742df5..b56e325 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -214,6 +214,7 @@ def backfill(args, dag=None):
delay_on_limit_secs=args.delay_on_limit,
verbose=args.verbose,
conf=run_conf,
+rerun_failed_tasks=args.rerun_failed_tasks,
)
@@ -1375,10 +1376,19 @@ class CLIFactory(object):
default=1.0),
'reset_dag_run': Arg(
("--reset_dagruns",),
-("if set, the backfill will delete existing "
- "backfill-related DAG runs and start "
- "anew with fresh, running DAG runs"),
+(
+"if set, the backfill will delete existing "
+"backfill-related DAG runs and start "
+"anew with fresh, running DAG runs"),
+"store_true"),
+'rerun_failed_tasks': Arg(
+("--rerun_failed_tasks",),
+(
+"if set, the backfill will auto-rerun "
+"all the failed tasks for the backfill date range "
+"instead of throwing exceptions"),
"store_true"),
+
# list_tasks
'tree': Arg(("-t", "--tree"), "Tree view", "store_true"),
# list_dags
@@ -1693,13 +1703,20 @@ class CLIFactory(object):
subparsers = (
{
'func': backfill,
-'help': "Run subsections of a DAG for a specified date range",
+'help': "Run subsections of a DAG for a specified date range. "
+"If reset_dag_run option is used,"
+" backfill will first prompt users whether airflow "
+"should clear all the previous dag_run and task_instances "
+"within the backfill date range."
+"If rerun_failed_tasks is used, backfill "
+"will auto re-run the previous failed task instances"
+" within the backfill date range.",
'args': (
'dag_id', 'task_regex', 'start_date', 'end_date',
'mark_success', 'local', 'donot_pickle',
'bf_ignore_dependencies', 'bf_ignore_first_depends_on_past',
'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose',
'conf',
-'reset_dag_run'
+'reset_dag_run', 'rerun_failed_tasks',
)
}, {
'func': list_tasks,
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e9f3fdc5/airflow/jobs.py
--
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 827349a..ad114ab 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1957,7 +1957,36 @@ class BackfillJob(BaseJob):
delay_on_limit_secs=1.0,
verbose=False,
conf=None,
+rerun_failed_tasks=False,
*args, **kwargs):
+"""
+:param dag: DAG object.
+:type dag: `class DAG`.
+:param start_date: start date for the backfill date range.
+:type start_date: datetime.
+:param end_date: end date for the backfill date range.
+:type end_date: datetime
+:param mark_success: flag whether to mark the task auto success.
+:type mark_success: bool
+:param donot_pickle: whether pickle
+:type donot_pickle: bool
+:param ignore_first_depends_on_past: whether to ignore depend on past
+:type ignore_first_depends_on_past: bool
+:param ignore_task_deps: whether to ignore the task dependency
+:type ignore_task_deps: bo
incubator-airflow git commit: [AIRFLOW-2558] Clear task/dag is clearing all executions
Repository: incubator-airflow
Updated Branches:
refs/heads/master 817296a7b -> c2b962ca9
[AIRFLOW-2558] Clear task/dag is clearing all executions
Closes #3465 from feng-tao/airflow_2588_new
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c2b962ca
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c2b962ca
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c2b962ca
Branch: refs/heads/master
Commit: c2b962ca9831cbf56f6dbb0147c97563b7ebbd8f
Parents: 817296a
Author: Tao feng
Authored: Tue Jun 5 15:14:43 2018 -0700
Committer: Maxime Beauchemin
Committed: Tue Jun 5 15:14:43 2018 -0700
--
airflow/bin/cli.py | 1 -
airflow/models.py | 43 ++-
tests/models.py| 4 ++--
3 files changed, 20 insertions(+), 28 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c2b962ca/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 3bff685..2742df5 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -199,7 +199,6 @@ def backfill(args, dag=None):
end_date=args.end_date,
confirm_prompt=True,
include_subdags=False,
-only_backfill_dagruns=True,
)
dag.run(
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c2b962ca/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index eda4808..c26ec01 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -132,17 +132,15 @@ def clear_task_instances(tis,
session,
activate_dag_runs=True,
dag=None,
- only_backfill_dagruns=False,
):
"""
Clears a set of task instances, but makes sure the running ones
-get killed. Reset backfill dag run state to removed if
only_backfill_dagruns is set
+get killed.
:param tis: a list of task instances
:param session: current session
:param activate_dag_runs: flag to check for active dag run
:param dag: DAG object
-:param only_backfill_dagruns: flag for setting backfill state
"""
job_ids = []
for ti in tis:
@@ -176,13 +174,8 @@ def clear_task_instances(tis,
DagRun.execution_date.in_({ti.execution_date for ti in tis}),
).all()
for dr in drs:
-if only_backfill_dagruns and dr.is_backfill:
-# If the flag is set, we reset backfill dag run for retry.
-# dont reset start date
-dr.state = State.REMOVED
-else:
-dr.state = State.RUNNING
-dr.start_date = timezone.utcnow()
+dr.state = State.RUNNING
+dr.start_date = timezone.utcnow()
class DagBag(BaseDagBag, LoggingMixin):
@@ -3697,18 +3690,20 @@ class DAG(BaseDag, LoggingMixin):
self,
state=State.RUNNING,
session=None,
-only_backfill_dagruns=False):
-drs = session.query(DagRun).filter_by(dag_id=self.dag_id).all()
+start_date=None,
+end_date=None,
+):
+query = session.query(DagRun).filter_by(dag_id=self.dag_id)
+if start_date:
+query = query.filter(DagRun.execution_date >= start_date)
+if end_date:
+query = query.filter(DagRun.execution_date <= end_date)
+drs = query.all()
+
dirty_ids = []
for dr in drs:
-if only_backfill_dagruns:
-if dr.is_backfill:
-dr.state = state
-dirty_ids.append(dr.dag_id)
-else:
-if not dr.is_backfill:
-dr.state = state
-dirty_ids.append(dr.dag_id)
+dr.state = state
+dirty_ids.append(dr.dag_id)
DagStat.update(dirty_ids, session=session)
@provide_session
@@ -3721,7 +3716,6 @@ class DAG(BaseDag, LoggingMixin):
reset_dag_runs=True,
dry_run=False,
session=None,
-only_backfill_dagruns=False,
):
"""
Clears a set of task instances associated with the current dag for
@@ -3772,11 +3766,12 @@ class DAG(BaseDag, LoggingMixin):
clear_task_instances(tis.all(),
session,
dag=self,
- only_backfill_dagruns=only_backfill_dagruns,
)
if reset_dag_runs:
self.set_dag_runs_
incubator-airflow git commit: [AIRFLOW-2537] Add reset-dagrun option to backfill command
Repository: incubator-airflow
Updated Branches:
refs/heads/master 2800c8e55 -> 4c6f1fdd6
[AIRFLOW-2537] Add reset-dagrun option to backfill command
Closes #3444 from feng-
tao/add_reset_dagrun_for_backfill
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/4c6f1fdd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/4c6f1fdd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/4c6f1fdd
Branch: refs/heads/master
Commit: 4c6f1fdd61a76a5eb7b284e2d0702339e842ef38
Parents: 2800c8e
Author: Tao feng
Authored: Fri Jun 1 11:26:23 2018 -0700
Committer: Maxime Beauchemin
Committed: Fri Jun 1 11:26:23 2018 -0700
--
airflow/bin/cli.py | 19 ++-
airflow/models.py | 62 ++---
tests/models.py| 38 +++---
3 files changed, 102 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/4c6f1fdd/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 10c104a..3bff685 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -192,6 +192,16 @@ def backfill(args, dag=None):
ti = TaskInstance(task, args.start_date)
ti.dry_run()
else:
+if args.reset_dagruns:
+DAG.clear_dags(
+[dag],
+start_date=args.start_date,
+end_date=args.end_date,
+confirm_prompt=True,
+include_subdags=False,
+only_backfill_dagruns=True,
+)
+
dag.run(
start_date=args.start_date,
end_date=args.end_date,
@@ -1364,6 +1374,12 @@ class CLIFactory(object):
"again."),
type=float,
default=1.0),
+'reset_dag_run': Arg(
+("--reset_dagruns",),
+("if set, the backfill will delete existing "
+ "backfill-related DAG runs and start "
+ "anew with fresh, running DAG runs"),
+"store_true"),
# list_tasks
'tree': Arg(("-t", "--tree"), "Tree view", "store_true"),
# list_dags
@@ -1683,7 +1699,8 @@ class CLIFactory(object):
'dag_id', 'task_regex', 'start_date', 'end_date',
'mark_success', 'local', 'donot_pickle',
'bf_ignore_dependencies', 'bf_ignore_first_depends_on_past',
-'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose',
'conf'
+'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose',
'conf',
+'reset_dag_run'
)
}, {
'func': list_tasks,
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/4c6f1fdd/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index e2794ab..eda4808 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -128,10 +128,21 @@ def get_fernet():
_CONTEXT_MANAGER_DAG = None
-def clear_task_instances(tis, session, activate_dag_runs=True, dag=None):
+def clear_task_instances(tis,
+ session,
+ activate_dag_runs=True,
+ dag=None,
+ only_backfill_dagruns=False,
+ ):
"""
Clears a set of task instances, but makes sure the running ones
-get killed.
+get killed. Reset backfill dag run state to removed if
only_backfill_dagruns is set
+
+:param tis: a list of task instances
+:param session: current session
+:param activate_dag_runs: flag to check for active dag run
+:param dag: DAG object
+:param only_backfill_dagruns: flag for setting backfill state
"""
job_ids = []
for ti in tis:
@@ -165,8 +176,13 @@ def clear_task_instances(tis, session,
activate_dag_runs=True, dag=None):
DagRun.execution_date.in_({ti.execution_date for ti in tis}),
).all()
for dr in drs:
-dr.state = State.RUNNING
-dr.start_date = timezone.utcnow()
+if only_backfill_dagruns and dr.is_backfill:
+# If the flag is set, we reset backfill dag run for retry.
+# dont reset start date
+dr.state = State.REMOVED
+else:
+dr.state = State.RUNNING
+dr.start_date = timezone.utcnow()
class DagBag(BaseDagBag, LoggingMixin):
@@ -3678,12 +3694,21 @@ class DAG(BaseDag, LoggingMixin):
@provide_session
def set_dag_runs_state(
-self, state=State.RUNNING, session=None):
-drs = session.query(DagModel).filter_by(d
incubator-airflow git commit: [AIRFLOW-2526] dag_run.conf can override params
Repository: incubator-airflow
Updated Branches:
refs/heads/master 8a5e5132d -> 2800c8e55
[AIRFLOW-2526] dag_run.conf can override params
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2526
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
params can be overridden by the dictionary passed
through `airflow backfill -c`
```
templated_command = """
echo "text = {{ params.text }}"
"""
bash_operator = BashOperator(
task_id='bash_task',
bash_command=templated_command,
dag=dag,
params= {
"text" : "normal processing"
})
```
In daily processing it prints:
```
normal processing
```
In backfill processing `airflow trigger_dag -c
"{"text": "override success"}"`, it prints
```
override success
```
### Tests
- [ ] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3422 from milton0825/params-overridden-
through-cli
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/2800c8e5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/2800c8e5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/2800c8e5
Branch: refs/heads/master
Commit: 2800c8e556401d68eb18036c11a5b6cb004ca7d5
Parents: 8a5e513
Author: Chao-Han Tsai
Authored: Fri Jun 1 11:22:10 2018 -0700
Committer: Maxime Beauchemin
Committed: Fri Jun 1 11:22:10 2018 -0700
--
airflow/config_templates/default_airflow.cfg | 4 +++
airflow/models.py| 7 +
docs/code.rst| 4 ++-
tests/models.py | 31 +++
4 files changed, 45 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2800c8e5/airflow/config_templates/default_airflow.cfg
--
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 9c7c96e..264757c 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -162,6 +162,10 @@ enable_xcom_pickling = True
# it has to cleanup after it is sent a SIGTERM, before it is SIGKILLED
killed_task_cleanup_time = 60
+# Whether to override params with dag_run.conf. If you pass some key-value
pairs through `airflow backfill -c` or
+# `airflow trigger_dag -c`, the key-value pairs will override the existing
ones in params.
+dag_run_conf_overrides_params = False
+
[cli]
# In what way should the cli access the API. The LocalClient will use the
# database directly, while the json_client will use the api running on the
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/2800c8e5/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 2511989..e2794ab 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1795,6 +1795,9 @@ class TaskInstance(Base, LoggingMixin):
if task.params:
params.update(task.params)
+if configuration.getboolean('core', 'dag_run_conf_overrides_params'):
+self.overwrite_params_with_dag_run_conf(params=params,
dag_run=dag_run)
+
class VariableAccessor:
"""
Wrapper around Variable. This way you can get variables in
templates by using
@@ -1862,6 +1865,10 @@ class TaskInstance(Base, LoggingMixin):
'outlets'
incubator-airflow git commit: [AIRFLOW-2538] Update faq doc on how to reduce airflow scheduler latency
Repository: incubator-airflow
Updated Branches:
refs/heads/master c6681681d -> b81bd08a3
[AIRFLOW-2538] Update faq doc on how to reduce airflow scheduler latency
Make sure you have checked _all_ steps below.
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2538
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
Update the faq doc on how to reduce airflow
scheduler latency. This comes from our internal
production setting which also aligns with Maxime's
email(https://lists.apache.org/thread.html/%3CCAHE
Ep7WFAivyMJZ0N+0Zd1T3nvfyCJRudL3XSRLM4utSigR3dQmai
l.gmail.com%3E).
### Tests
- [ ] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
### Commits
- [ ] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [ ] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [ ] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3434 from feng-tao/update_faq
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b81bd08a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b81bd08a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b81bd08a
Branch: refs/heads/master
Commit: b81bd08a334efa5242af705743519be43346295e
Parents: c668168
Author: Tao feng
Authored: Thu May 31 22:01:59 2018 -0700
Committer: Maxime Beauchemin
Committed: Thu May 31 22:01:59 2018 -0700
--
docs/faq.rst | 10 +-
1 file changed, 9 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b81bd08a/docs/faq.rst
--
diff --git a/docs/faq.rst b/docs/faq.rst
index d2c6188..33b4d5a 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -162,10 +162,18 @@ How can we reduce the airflow UI page load time?
If your dag takes long time to load, you could reduce the value of
``default_dag_run_display_number`` configuration in ``airflow.cfg`` to a
smaller value. This configurable controls the number of dag run to show in UI
with default value 25.
+
How to fix Exception: Global variable explicit_defaults_for_timestamp needs to
be on (1)?
--
+-
This means ``explicit_defaults_for_timestamp`` is disabled in your mysql
server and you need to enable it by:
#. Set ``explicit_defaults_for_timestamp = 1`` under the mysqld section in
your my.cnf file.
#. Restart the Mysql server.
+
+
+How to reduce airflow dag scheduling latency in production?
+---
+
+- ``max_threads``: Scheduler will spawn multiple threads in parallel to
schedule dags. This is controlled by ``max_threads`` with default value of 2.
User should increase this value to a larger value(e.g numbers of cpus where
scheduler runs - 1) in production.
+- ``scheduler_heartbeat_sec``: User should consider to increase
``scheduler_heartbeat_sec`` config to a higher value(e.g 60 secs) which
controls how frequent the airflow scheduler gets the heartbeat and updates the
job's entry in database.
incubator-airflow git commit: [AIRFLOW-2517] backfill support passing key values through CLI
Repository: incubator-airflow
Updated Branches:
refs/heads/master 7945854cc -> 3ed25a945
[AIRFLOW-2517] backfill support passing key values through CLI
### JIRA
- [x] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "\[AIRFLOW-XXX\] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-2517
- In case you are fixing a typo in the
documentation you can prepend your commit with
\[AIRFLOW-XXX\], code changes always need a JIRA
issue.
### Description
- [x] Here are some details about my PR, including
screenshots of any UI changes:
In backfill, we can provide key-value pairs
through CLI and those pairs can be accessed
through macros. This is just like the way
`trigger_dag -c` works [1].
Let's walk through an example.
In the airflow CLI we specify a key-value pair.
```
airflow backfill hello_world -s 2018-02-01 -e
2018-02-08 -c '{"text": "some text"}'
```
In the DAG file, I have a `BashOperator` that
contains a template command and I want
{{ dag_run.conf.text }} resolves to the text I
passed in CLI.
```python
templated_command = """
echo "ds = {{ ds }}"
echo "prev_ds = {{
macros.datetime.strftime(prev_execution_date,
"%Y-%m-%d") }}"
echo "next_ds = {{
macros.datetime.strftime(next_execution_date,
"%Y-%m-%d") }}"
echo "text_through_conf = {{ dag_run.conf.text }}"
"""
bash_operator = BashOperator(
task_id='bash_task',
bash_command=templated_command,
dag=dag
)
```
Rendered Bash command in Airflow UI.
https://user-images.githubuserconte
nt.com/6065051/40395666-04c41574-5dde-11e8-9ec2-c0
312b7203e6.png">
[1]
https://airflow.apache.org/cli.html#trigger_dag
### Tests
- [x] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
### Commits
- [x] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
### Documentation
- [x] In case of new functionality, my PR adds
documentation that describes how to use it.
- When adding new operators/hooks/sensors, the
autoclass documentation generation needs to be
added.
### Code Quality
- [x] Passes `git diff upstream/master -u --
"*.py" | flake8 --diff`
Closes #3406 from milton0825/backfill-support-conf
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3ed25a94
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3ed25a94
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3ed25a94
Branch: refs/heads/master
Commit: 3ed25a9459a2a0b26b849b1a3ea7906f26a33707
Parents: 7945854
Author: Chao-Han Tsai
Authored: Wed May 30 10:50:06 2018 -0700
Committer: Maxime Beauchemin
Committed: Wed May 30 10:50:06 2018 -0700
--
airflow/bin/cli.py | 7 ++-
airflow/jobs.py| 5 -
airflow/models.py | 4
tests/jobs.py | 28
4 files changed, 42 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3ed25a94/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 55201dd..10c104a 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -180,6 +180,10 @@ def backfill(args, dag=None):
task_regex=args.task_regex,
include_upstream=not args.ignore_dependencies)
+run_conf = None
+if args.conf:
+run_conf = json.loads(args.conf)
+
if args.dry_run:
print("Dry run of DAG {0} on {1}".format(args.dag_id,
args.start_date))
@@ -200,6 +204,7 @@ def backfill(args, dag=None):
pool=args.pool,
delay_on_limit_secs=args.delay_on_limit,
verbose=args.verbose,
+conf=run_conf,
)
@@ -1678,7 +1683,7 @@ class CLIFactory(object):
'dag_id', 'task_regex', 'start_date', 'end_date',
'mark_success', 'local', 'donot_pickle',
'bf_ignore_dependencies', 'bf_ignore_first_depends_on_past',
-'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose',
+'subdir', 'pool', 'delay_on_limit', 'dry_run', 'verbose',
'conf'
)
}, {
incubator-airflow git commit: [AIRFLOW-2419] Use default view for subdag operator
Repository: incubator-airflow
Updated Branches:
refs/heads/master 62b95f8bc -> f1ac67bdc
[AIRFLOW-2419] Use default view for subdag operator
Closes #3314 from milanvdm/milanvdm/subdag_view
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f1ac67bd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f1ac67bd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f1ac67bd
Branch: refs/heads/master
Commit: f1ac67bdc6649944d34e6568ae64c6a376dc22cd
Parents: 62b95f8
Author: milanvdm
Authored: Wed May 23 11:46:13 2018 -0700
Committer: Maxime Beauchemin
Committed: Wed May 23 11:46:13 2018 -0700
--
airflow/www/templates/airflow/dag.html | 4 ++--
airflow/www_rbac/templates/airflow/dag.html | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f1ac67bd/airflow/www/templates/airflow/dag.html
--
diff --git a/airflow/www/templates/airflow/dag.html
b/airflow/www/templates/airflow/dag.html
index 18242d3..adb2d38 100644
--- a/airflow/www/templates/airflow/dag.html
+++ b/airflow/www/templates/airflow/dag.html
@@ -49,7 +49,7 @@
{% if dag.parent_dag %}
-
+
Back to {{ dag.parent_dag.dag_id }}
@@ -311,7 +311,7 @@ function updateQueryStringParameter(uri, key, value) {
});
$("#btn_subdag").click(function(){
- url = "{{ url_for('airflow.graph') }}" +
+ url = "{{ url_for( 'airflow.' + dag.default_view ) }}" +
"?dag_id=" + encodeURIComponent(subdag_id) +
"&execution_date=" + encodeURIComponent(execution_date);
window.location = url;
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f1ac67bd/airflow/www_rbac/templates/airflow/dag.html
--
diff --git a/airflow/www_rbac/templates/airflow/dag.html
b/airflow/www_rbac/templates/airflow/dag.html
index 35308be..eb44908 100644
--- a/airflow/www_rbac/templates/airflow/dag.html
+++ b/airflow/www_rbac/templates/airflow/dag.html
@@ -48,7 +48,7 @@
{% if dag.parent_dag %}
-
+
Back to {{ dag.parent_dag.dag_id }}
@@ -309,7 +309,7 @@ function updateQueryStringParameter(uri, key, value) {
});
$("#btn_subdag").click(function(){
- url = "{{ url_for('Airflow.graph') }}" +
+ url = "{{ url_for( 'Airflow.' + dag.default_view ) }}" +
"?dag_id=" + encodeURIComponent(subdag_id) +
"&execution_date=" + encodeURIComponent(execution_date);
window.location = url;
incubator-airflow git commit: [AIRFLOW-2442][AIRFLOW-2] Airflow run command leaves database connections open
Repository: incubator-airflow
Updated Branches:
refs/heads/master 961aa0581 -> 250faad0f
[AIRFLOW-2442][AIRFLOW-2] Airflow run command leaves database connections open
Closes #3336 from afernandez/afernandez_db
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/250faad0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/250faad0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/250faad0
Branch: refs/heads/master
Commit: 250faad0f557bb8deac8bd0b948112bcaf48004a
Parents: 961aa05
Author: Alejandro Fernandez
Authored: Thu May 10 15:29:33 2018 -0700
Committer: Maxime Beauchemin
Committed: Thu May 10 15:29:33 2018 -0700
--
airflow/bin/cli.py | 15 +++
airflow/config_templates/default_airflow.cfg | 5 ++--
airflow/settings.py | 31 ++-
3 files changed, 38 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/250faad0/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 3df8166..1615e76 100644
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -419,10 +419,6 @@ def _run(args, dag, ti):
@cli_utils.action_logging
def run(args, dag=None):
-# Disable connection pooling to reduce the # of connections on the DB
-# while it's waiting for the task to finish.
-settings.configure_orm(disable_connection_pool=True)
-
if dag:
args.dag_id = dag.dag_id
@@ -436,11 +432,20 @@ def run(args, dag=None):
if os.path.exists(args.cfg_path):
os.remove(args.cfg_path)
+# Do not log these properties since some may contain passwords.
+# This may also set default values for database properties like
+# core.sql_alchemy_pool_size
+# core.sql_alchemy_pool_recycle
for section, config in conf_dict.items():
for option, value in config.items():
conf.set(section, option, value)
settings.configure_vars()
-settings.configure_orm()
+
+# IMPORTANT, have to use the NullPool, otherwise, each "run" command may
leave
+# behind multiple open sleeping connections while heartbeating, which could
+# easily exceed the database connection limit when
+# processing hundreds of simultaneous tasks.
+settings.configure_orm(disable_connection_pool=True)
if not args.pickle and not dag:
dag = get_dag(args)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/250faad0/airflow/config_templates/default_airflow.cfg
--
diff --git a/airflow/config_templates/default_airflow.cfg
b/airflow/config_templates/default_airflow.cfg
index 33b99ff..d2e77a7 100644
--- a/airflow/config_templates/default_airflow.cfg
+++ b/airflow/config_templates/default_airflow.cfg
@@ -89,8 +89,9 @@ sql_alchemy_pool_size = 5
# The SqlAlchemy pool recycle is the number of seconds a connection
# can be idle in the pool before it is invalidated. This config does
-# not apply to sqlite.
-sql_alchemy_pool_recycle = 3600
+# not apply to sqlite. If the number of DB connections is ever exceeded,
+# a lower config value will allow the system to recover faster.
+sql_alchemy_pool_recycle = 1800
# How many seconds to retry re-establishing a DB connection after
# disconnects. Setting this to 0 disables retries.
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/250faad0/airflow/settings.py
--
diff --git a/airflow/settings.py b/airflow/settings.py
index 3ee0d5d..2abe568 100644
--- a/airflow/settings.py
+++ b/airflow/settings.py
@@ -7,9 +7,9 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
-#
+#
# http://www.apache.org/licenses/LICENSE-2.0
-#
+#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -151,11 +151,29 @@ def configure_orm(disable_connection_pool=False):
pool_connections = conf.getboolean('core', 'SQL_ALCHEMY_POOL_ENABLED')
if disable_connection_pool or not pool_connections:
engine_args['poolclass'] = NullPool
+log.info("settings.configure_orm(): Using NullPool")
elif 'sqlite' not in SQL_ALCHEMY_CONN:
-# Engine args not supported by sqlite
-engine_args['pool_size'] = conf.getint('core', 'SQL_ALCHEMY_POOL_SIZE')
-engine_args['pool_recycle'] = conf.getint('core',
-
incubator-airflow git commit: [AIRFLOW-2253] Add Airflow CLI instrumentation
Repository: incubator-airflow
Updated Branches:
refs/heads/master a6487d619 -> 9dba430b6
[AIRFLOW-2253] Add Airflow CLI instrumentation
Closes #3159 from jinhyukchang/cli-instrumentation
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9dba430b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9dba430b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9dba430b
Branch: refs/heads/master
Commit: 9dba430b683361fc0ed7f50de6daa03c971a476b
Parents: a6487d6
Author: Jin Hyuk Chang
Authored: Mon Apr 2 15:16:25 2018 -0700
Committer: Maxime Beauchemin
Committed: Mon Apr 2 15:16:25 2018 -0700
--
airflow/bin/cli.py | 30
airflow/settings.py | 11 ++-
airflow/utils/cli.py| 114 +++
airflow/utils/cli_action_loggers.py | 101 +++
tests/utils/test_cli_util.py| 97 ++
5 files changed, 352 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9dba430b/airflow/bin/cli.py
--
diff --git a/airflow/bin/cli.py b/airflow/bin/cli.py
index 24f5d31..eb96e77 100755
--- a/airflow/bin/cli.py
+++ b/airflow/bin/cli.py
@@ -54,6 +54,7 @@ from airflow.models import (DagModel, DagBag, TaskInstance,
Connection, DAG)
from airflow.ti_deps.dep_context import (DepContext, SCHEDULER_DEPS)
+from airflow.utils import cli as cli_utils
from airflow.utils import db as db_utils
from airflow.utils.net import get_hostname
from airflow.utils.log.logging_mixin import (LoggingMixin, redirect_stderr,
@@ -149,6 +150,7 @@ def get_dags(args):
return matched_dags
+@cli_utils.action_logging
def backfill(args, dag=None):
logging.basicConfig(
level=settings.LOGGING_LEVEL,
@@ -190,6 +192,7 @@ def backfill(args, dag=None):
delay_on_limit_secs=args.delay_on_limit)
+@cli_utils.action_logging
def trigger_dag(args):
"""
Creates a dag run for the specified dag
@@ -208,6 +211,7 @@ def trigger_dag(args):
log.info(message)
+@cli_utils.action_logging
def delete_dag(args):
"""
Deletes all DB records related to the specified dag
@@ -228,6 +232,7 @@ def delete_dag(args):
print("Bail.")
+@cli_utils.action_logging
def pool(args):
log = LoggingMixin().log
@@ -252,6 +257,7 @@ def pool(args):
log.info(_tabulate(pools=pools))
+@cli_utils.action_logging
def variables(args):
if args.get:
try:
@@ -328,10 +334,12 @@ def export_helper(filepath):
print("{} variables successfully exported to {}".format(len(var_dict),
filepath))
+@cli_utils.action_logging
def pause(args, dag=None):
set_is_paused(True, args, dag)
+@cli_utils.action_logging
def unpause(args, dag=None):
set_is_paused(False, args, dag)
@@ -401,6 +409,7 @@ def _run(args, dag, ti):
executor.end()
+@cli_utils.action_logging
def run(args, dag=None):
# Disable connection pooling to reduce the # of connections on the DB
# while it's waiting for the task to finish.
@@ -453,6 +462,8 @@ def run(args, dag=None):
_run(args, dag, ti)
logging.shutdown()
+
+@cli_utils.action_logging
def task_failed_deps(args):
"""
Returns the unmet dependencies for a task instance from the perspective of
the
@@ -479,6 +490,7 @@ def task_failed_deps(args):
print("Task instance dependencies are all met.")
+@cli_utils.action_logging
def task_state(args):
"""
Returns the state of a TaskInstance at the command line.
@@ -492,6 +504,7 @@ def task_state(args):
print(ti.current_state())
+@cli_utils.action_logging
def dag_state(args):
"""
Returns the state of a DagRun at the command line.
@@ -504,6 +517,7 @@ def dag_state(args):
print(dr[0].state if len(dr) > 0 else None)
+@cli_utils.action_logging
def list_dags(args):
dagbag = DagBag(process_subdir(args.subdir))
s = textwrap.dedent("""\n
@@ -518,6 +532,7 @@ def list_dags(args):
print(dagbag.dagbag_report())
+@cli_utils.action_logging
def list_tasks(args, dag=None):
dag = dag or get_dag(args)
if args.tree:
@@ -527,6 +542,7 @@ def list_tasks(args, dag=None):
print("\n".join(sorted(tasks)))
+@cli_utils.action_logging
def test(args, dag=None):
dag = dag or get_dag(args)
@@ -543,6 +559,7 @@ def test(args, dag=None):
ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
+@cli_utils.action_logging
def render(args):
dag = get_dag(args)
task = dag.get_task(task_id=args.task_id)
@@ -557,6 +574,7 @@ def render(args):
incubator-airflow git commit: [AIRFLOW-1460] Allow restoration of REMOVED TI's
Repository: incubator-airflow
Updated Branches:
refs/heads/master 8754cb1c6 -> 8c42d03c4
[AIRFLOW-1460] Allow restoration of REMOVED TI's
When a task instance exists in the database but
its corresponding task
no longer exists in the DAG, the scheduler marks
the task instance as
REMOVED. Once removed, task instances stayed
removed forever, even if
the task were to be added back to the DAG.
This change allows for the restoration of REMOVED
task instances. If a
task instance is in state REMOVED but the
corresponding task is present
in the DAG, restore the task instance by setting
its state to NONE.
A new unit test simulates the removal and
restoration of a task from a
DAG and verifies that the task instance is
restored:
`./run_unit_tests.sh tests.models:DagRunTest`
JIRA:
https://issues.apache.org/jira/browse/AIRFLOW-1460
Closes #3137 from astahlman/airflow-1460-restore-
tis
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/8c42d03c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/8c42d03c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/8c42d03c
Branch: refs/heads/master
Commit: 8c42d03c4e35a0046e46f0e2e6db588702ee7e8b
Parents: 8754cb1
Author: Andrew Stahlman
Authored: Wed Mar 21 23:54:05 2018 -0700
Committer: Maxime Beauchemin
Committed: Wed Mar 21 23:54:05 2018 -0700
--
airflow/models.py | 21 ++---
tests/models.py | 24
2 files changed, 42 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8c42d03c/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index c1b608a..aa10ad5 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -4905,16 +4905,31 @@ class DagRun(Base, LoggingMixin):
dag = self.get_dag()
tis = self.get_task_instances(session=session)
-# check for removed tasks
+# check for removed or restored tasks
task_ids = []
for ti in tis:
task_ids.append(ti.task_id)
+task = None
try:
-dag.get_task(ti.task_id)
+task = dag.get_task(ti.task_id)
except AirflowException:
-if self.state is not State.RUNNING and not dag.partial:
+if ti.state == State.REMOVED:
+pass # ti has already been removed, just ignore it
+elif self.state is not State.RUNNING and not dag.partial:
+self.log.warning("Failed to get task '{}' for dag '{}'. "
+ "Marking it as removed.".format(ti, dag))
+Stats.incr(
+"task_removed_from_dag.{}".format(dag.dag_id), 1, 1)
ti.state = State.REMOVED
+is_task_in_dag = task is not None
+should_restore_task = is_task_in_dag and ti.state == State.REMOVED
+if should_restore_task:
+self.log.info("Restoring task '{}' which was previously "
+ "removed from DAG '{}'".format(ti, dag))
+Stats.incr("task_restored_to_dag.{}".format(dag.dag_id), 1, 1)
+ti.state = State.NONE
+
# check for missing tasks
for task in six.itervalues(dag.task_dict):
if task.adhoc:
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/8c42d03c/tests/models.py
--
diff --git a/tests/models.py b/tests/models.py
index 5d8184c..98913af 100644
--- a/tests/models.py
+++ b/tests/models.py
@@ -892,6 +892,30 @@ class DagRunTest(unittest.TestCase):
self.assertTrue(dagrun.is_backfill)
self.assertFalse(dagrun2.is_backfill)
+def test_removed_task_instances_can_be_restored(self):
+def with_all_tasks_removed(dag):
+return DAG(dag_id=dag.dag_id, start_date=dag.start_date)
+
+dag = DAG('test_task_restoration', start_date=DEFAULT_DATE)
+dag.add_task(DummyOperator(task_id='flaky_task', owner='test'))
+
+dagrun = self.create_dag_run(dag)
+flaky_ti = dagrun.get_task_instances()[0]
+self.assertEquals('flaky_task', flaky_ti.task_id)
+self.assertEquals(State.NONE, flaky_ti.state)
+
+dagrun.dag = with_all_tasks_removed(dag)
+
+dagrun.verify_integrity()
+flaky_ti.refresh_from_db()
+self.assertEquals(State.REMOVED, flaky_ti.state)
+
+dagrun.dag.add_task(DummyOperator(task_id='flaky_task', owner='test'))
+
+dagrun.verify_integrity()
+flaky_ti.refresh_from_db()
+self.assertEquals(State.NONE, flaky_ti.state)
+
class DagBag
svn commit: r21442 - /release/incubator/airflow/1.8.2-incubating/
Author: maximebeauchemin Date: Mon Sep 4 17:20:21 2017 New Revision: 21442 Log: Releasing Apache Airflow 1.8.2-incubating in the proper folder Added: release/incubator/airflow/1.8.2-incubating/ release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz (with props) release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.asc release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.sha release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz (with props) release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.asc release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.md5 release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.sha Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz == Binary file - no diff available. Propchange: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz -- svn:mime-type = application/octet-stream Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.asc == --- release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.asc (added) +++ release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.asc Mon Sep 4 17:20:21 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmJRgsACgkQ5vBQXMe8 +fg2OXAf+Ka04u5a9lX+sALgZ9Hsw5q/VQJBIIWmVigIEvhoNxcX61lzCbHy/TvAF +fYRQ2dQC69XlGldFmDwZ6Ojf/oBzRgP/AqA2JPIEBre+FdEUNXlLQ2jk38zI4pwR +3lXnMMSgGvTwZ2mwNPnFJ7OyEv5GMwg01nJyT9r0kPv86KCqOXSI8dbv7uSrPHqn +CmME+seUGBodL7vCNpzNuFeuYPt/QldbAo2rSBHJpIWoE8Vo9zxdFncGdw06UKs6 +BMtmUVv2NLWCrZdUgbm3bk9yfsTmew6JZbRt1Rdx4CWwHNDS98Zect+20koVl06y +ChuSrnz1CBRsXNNz5eiSbGSS4GHGkw== +=On8R +-END PGP SIGNATURE- Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 == --- release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 (added) +++ release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 Mon Sep 4 17:20:21 2017 @@ -0,0 +1,2 @@ +apache-airflow-1.8.2+incubating-bin.tar.gz: +7E 57 ED A7 14 84 7F 00 57 F3 E3 1D AF 90 A3 D6 Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.sha == --- release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.sha (added) +++ release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-bin.tar.gz.sha Mon Sep 4 17:20:21 2017 @@ -0,0 +1,3 @@ +apache-airflow-1.8.2+incubating-bin.tar.gz: +F32FFE95 BDC2066A 125624F1 62731E34 AD45D5A4 01B73345 FDD3FF09 FFD9CEF8 632611DF + B943FA79 9074B8AA 1B54616F EB47A3DB B35D740A 1DBB907D 32E83F59 Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz == Binary file - no diff available. Propchange: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz -- svn:mime-type = application/octet-stream Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.asc == --- release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.asc (added) +++ release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.asc Mon Sep 4 17:20:21 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmJRdsACgkQ5vBQXMe8 +fg24Lgf/YdsvHcwKJXD3qb2V4MshGEN+oLeDwl70dfiwqgMksBh2Pv90yaS+9pkG +zhbIXJyCFuIKxMV6Sm70P2/Ky3XkAx81HsTq/pO5dsHeVC+IgXCkWnPHkWUag4lK +jT8RKBnQDB376ofTNO5caIzpK8fc4ngYNHqEnnnVPi32Dvp9wE7E1nC/xxPtbt7K +WoTr484w4F2rIy6a477Cgkoazf2ivdKrfMdp9xihqcYMBY1agTY2eCzfVBR6vFoL +xWQV8NVPxHUNbRCZpuBKa/2a/huhv+dvIfdR5P+P21QWtSVJeNgkMobU8cvbriS/ +KehC0SSR493mqsZlbSvPTB/zBSbl8g== +=3GR+ +-END PGP SIGNATURE- Added: release/incubator/airflow/1.8.2-incubating/apache-airflow-1.8.2+incubating-source.tar.gz.md5
svn commit: r21441 - /release/airflow/
Author: maximebeauchemin Date: Mon Sep 4 17:17:23 2017 New Revision: 21441 Log: Correcting folder location of Airflow 1.8.2 Removed: release/airflow/
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.8.2 [created] 32a26d84b
svn commit: r21440 - /release/airflow/1.8.2/
Author: maximebeauchemin Date: Mon Sep 4 16:47:56 2017 New Revision: 21440 Log: Releasing Apache Airflow 1.8.2 Added: release/airflow/1.8.2/ release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz (with props) release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.asc release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.sha release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz (with props) release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.asc release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.md5 release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.sha Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz == Binary file - no diff available. Propchange: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz -- svn:mime-type = application/octet-stream Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.asc == --- release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.asc (added) +++ release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.asc Mon Sep 4 16:47:56 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmJRgsACgkQ5vBQXMe8 +fg2OXAf+Ka04u5a9lX+sALgZ9Hsw5q/VQJBIIWmVigIEvhoNxcX61lzCbHy/TvAF +fYRQ2dQC69XlGldFmDwZ6Ojf/oBzRgP/AqA2JPIEBre+FdEUNXlLQ2jk38zI4pwR +3lXnMMSgGvTwZ2mwNPnFJ7OyEv5GMwg01nJyT9r0kPv86KCqOXSI8dbv7uSrPHqn +CmME+seUGBodL7vCNpzNuFeuYPt/QldbAo2rSBHJpIWoE8Vo9zxdFncGdw06UKs6 +BMtmUVv2NLWCrZdUgbm3bk9yfsTmew6JZbRt1Rdx4CWwHNDS98Zect+20koVl06y +ChuSrnz1CBRsXNNz5eiSbGSS4GHGkw== +=On8R +-END PGP SIGNATURE- Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 == --- release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 (added) +++ release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 Mon Sep 4 16:47:56 2017 @@ -0,0 +1,2 @@ +apache-airflow-1.8.2+incubating-bin.tar.gz: +7E 57 ED A7 14 84 7F 00 57 F3 E3 1D AF 90 A3 D6 Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.sha == --- release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.sha (added) +++ release/airflow/1.8.2/apache-airflow-1.8.2+incubating-bin.tar.gz.sha Mon Sep 4 16:47:56 2017 @@ -0,0 +1,3 @@ +apache-airflow-1.8.2+incubating-bin.tar.gz: +F32FFE95 BDC2066A 125624F1 62731E34 AD45D5A4 01B73345 FDD3FF09 FFD9CEF8 632611DF + B943FA79 9074B8AA 1B54616F EB47A3DB B35D740A 1DBB907D 32E83F59 Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz == Binary file - no diff available. Propchange: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz -- svn:mime-type = application/octet-stream Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.asc == --- release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.asc (added) +++ release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.asc Mon Sep 4 16:47:56 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmJRdsACgkQ5vBQXMe8 +fg24Lgf/YdsvHcwKJXD3qb2V4MshGEN+oLeDwl70dfiwqgMksBh2Pv90yaS+9pkG +zhbIXJyCFuIKxMV6Sm70P2/Ky3XkAx81HsTq/pO5dsHeVC+IgXCkWnPHkWUag4lK +jT8RKBnQDB376ofTNO5caIzpK8fc4ngYNHqEnnnVPi32Dvp9wE7E1nC/xxPtbt7K +WoTr484w4F2rIy6a477Cgkoazf2ivdKrfMdp9xihqcYMBY1agTY2eCzfVBR6vFoL +xWQV8NVPxHUNbRCZpuBKa/2a/huhv+dvIfdR5P+P21QWtSVJeNgkMobU8cvbriS/ +KehC0SSR493mqsZlbSvPTB/zBSbl8g== +=3GR+ +-END PGP SIGNATURE- Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.md5 == --- release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.md5 (added) +++ release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.md5 Mon Sep 4 16:47:56 2017 @@ -0,0 +1,2 @@ +apache-airflow-1.8.2+incubating-source.tar.gz: +62 D3 71 C2 E8 28 F6 63 1E 8D 86 46 F0 9B F5 93 Added: release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.sha == --- release/airflow/1.8.2/apache-airflow-1.8.2+incubating-source.tar.gz.sha (added) +++ release/airflow/1.8.2/apache
svn commit: r21439 - /release/airflow/
Author: maximebeauchemin Date: Mon Sep 4 16:42:27 2017 New Revision: 21439 Log: Adding a folder for Apache Airflow Added: release/airflow/
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.8.2rc4 [created] 32a26d84b
svn commit: r20895 - in /dev/incubator/airflow: ./ 1.8.2rc4/
Author: maximebeauchemin Date: Tue Aug 8 05:08:46 2017 New Revision: 20895 Log: 1.8.2rc4 Added: dev/incubator/airflow/1.8.2rc4/ dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz (with props) dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.asc dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.sha dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz (with props) dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.asc dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.md5 dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.sha Removed: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.sha Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz == Binary file - no diff available. Propchange: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.asc == --- dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.asc (added) +++ dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.asc Tue Aug 8 05:08:46 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmJRgsACgkQ5vBQXMe8 +fg2OXAf+Ka04u5a9lX+sALgZ9Hsw5q/VQJBIIWmVigIEvhoNxcX61lzCbHy/TvAF +fYRQ2dQC69XlGldFmDwZ6Ojf/oBzRgP/AqA2JPIEBre+FdEUNXlLQ2jk38zI4pwR +3lXnMMSgGvTwZ2mwNPnFJ7OyEv5GMwg01nJyT9r0kPv86KCqOXSI8dbv7uSrPHqn +CmME+seUGBodL7vCNpzNuFeuYPt/QldbAo2rSBHJpIWoE8Vo9zxdFncGdw06UKs6 +BMtmUVv2NLWCrZdUgbm3bk9yfsTmew6JZbRt1Rdx4CWwHNDS98Zect+20koVl06y +ChuSrnz1CBRsXNNz5eiSbGSS4GHGkw== +=On8R +-END PGP SIGNATURE- Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 == --- dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 (added) +++ dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.md5 Tue Aug 8 05:08:46 2017 @@ -0,0 +1,2 @@ +apache-airflow-1.8.2+incubating-bin.tar.gz: +7E 57 ED A7 14 84 7F 00 57 F3 E3 1D AF 90 A3 D6 Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.sha == --- dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.sha (added) +++ dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-bin.tar.gz.sha Tue Aug 8 05:08:46 2017 @@ -0,0 +1,3 @@ +apache-airflow-1.8.2+incubating-bin.tar.gz: +F32FFE95 BDC2066A 125624F1 62731E34 AD45D5A4 01B73345 FDD3FF09 FFD9CEF8 632611DF + B943FA79 9074B8AA 1B54616F EB47A3DB B35D740A 1DBB907D 32E83F59 Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz == Binary file - no diff available. Propchange: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.asc == --- dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.asc (added) +++ dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.asc Tue Aug 8 05:08:46 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmJRdsACgkQ5vBQXMe8 +fg24Lgf/YdsvHcwKJXD3qb2V4MshGEN+oLeDwl70dfiwqgMksBh2Pv90yaS+9pkG +zhbIXJyCFuIKxMV6Sm70P2/Ky3XkAx81HsTq/pO5dsHeVC+IgXCkWnPHkWUag4lK +jT8RKBnQDB376ofTNO5caIzpK8fc4ngYNHqEnnnVPi32Dvp9wE7E1nC/xxPtbt7K +WoTr484w4F2rIy6a477Cgkoazf2ivdKrfMdp9xihqcYMBY1agTY2eCzfVBR6vFoL +xWQV8NVPxHUNbRCZpuBKa/2a/huhv+dvIfdR5P+P21QWtSVJeNgkMobU8cvbriS/ +KehC0SSR493mqsZlbSvPTB/zBSbl8g== +=3GR+ +-END PGP SIGNATURE- Added: dev/incubator/airflow/1.8.2rc4/apache-airflow-1.8.2+incubating-source.tar.gz.md5 == --- dev/incubator/airflow/1.8.2rc4/apache-airflow
incubator-airflow git commit: Set version 1.8.2rc2 -> 1.8.2
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test 0be35d628 -> 32a26d84b Set version 1.8.2rc2 -> 1.8.2 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/32a26d84 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/32a26d84 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/32a26d84 Branch: refs/heads/v1-8-test Commit: 32a26d84b679a54add43092d0bdb77350dcbaeaf Parents: 0be35d6 Author: Maxime Beauchemin Authored: Mon Aug 7 21:57:29 2017 -0700 Committer: Maxime Beauchemin Committed: Mon Aug 7 21:57:29 2017 -0700 -- airflow/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/32a26d84/airflow/version.py -- diff --git a/airflow/version.py b/airflow/version.py index ce57c88..66b4c60 100644 --- a/airflow/version.py +++ b/airflow/version.py @@ -13,4 +13,4 @@ # limitations under the License. # -version = '1.8.2rc2' +version = '1.8.2'
svn commit: r20705 - /dev/incubator/airflow/
Author: maximebeauchemin Date: Tue Aug 1 19:58:46 2017 New Revision: 20705 Log: Pushing rc3 Added: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz (with props) dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.sha Removed: dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.sha Added: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz == Binary file - no diff available. Propchange: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.asc == --- dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.asc (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.asc Tue Aug 1 19:58:46 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAlmA2e4ACgkQ5vBQXMe8 +fg0RcAf/Vx1stsx9/PuYX2JNw5oERC4YS/8b0G4hUkA3ZPdesh9DssYyMeNip7BY +eDx9xo6olcf3GhvsvZxhT//V+waqjsgyJlsOKMPUl07s6+U6lvIsjowahISeMaqb +QFMOQPGagkjXFouAMi2t7d7SgLiSk4EqAf7DKAW4G5lt55AU3ZyrXsIQpdj2STDe +DV0mvRYO7wcqOf7Wj8a3SKPmeB+bYq4+UJwd//SNMZrKWGLCwnTwSS2Z0D0VGY+A +nrmZOdJ1uEe4ZJ4/rRvt0heCmDE//ZH9DSzEhNXm8oDh7IL4TJmJwOfTlnPVEjji +xVcdOtHQ0xI4lfIZ3Oi+XJ+bAopTiA== +=OFoj +-END PGP SIGNATURE- Added: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.md5 == --- dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.md5 (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.md5 Tue Aug 1 19:58:46 2017 @@ -0,0 +1,2 @@ +apache-airflow-1.8.2rc3+incubating.tar.gz: +FA F0 19 33 D2 76 71 A5 82 F9 D9 0D 97 4B 30 FD Added: dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.sha == --- dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.sha (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc3+incubating.tar.gz.sha Tue Aug 1 19:58:46 2017 @@ -0,0 +1,3 @@ +apache-airflow-1.8.2rc3+incubating.tar.gz: +A9A53A18 CC7115DC E09847B0 39F3EA1D 58B3CF4B 097EBFBA 0AE0E0E8 E2F8CDCD 7BFF345C + 71414EA4 493AF080 97BB7705 52BBF96E BD94ACC9 0E062EFA 571E36C1
incubator-airflow git commit: [AIRFLOW-1476] add INSTALL instruction for source releases
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test 302520828 -> 0be35d628 Updated Tags: refs/tags/1.8.2rc3 [created] 0be35d628 [AIRFLOW-1476] add INSTALL instruction for source releases Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/0be35d62 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/0be35d62 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/0be35d62 Branch: refs/heads/v1-8-test Commit: 0be35d6280cd3e2a64b02989de4a3d99e24f0989 Parents: 3025208 Author: Maxime Beauchemin Authored: Mon Jul 31 15:48:41 2017 -0700 Committer: Maxime Beauchemin Committed: Tue Aug 1 12:10:38 2017 -0700 -- .rat-excludes | 10 ++ INSTALL | 9 + 2 files changed, 19 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0be35d62/.rat-excludes -- diff --git a/.rat-excludes b/.rat-excludes index d5b8b4e..725a305 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -25,3 +25,13 @@ CHANGELOG.txt kerberos_auth.py airflow_api_auth_backend_kerberos_auth_py.html licenses/* +airflow/www/static/docs +parallel.js +underscore.js +jquery.dataTables.min.js +jqClock.min.js +dagre-d3.min.js +bootstrap-toggle.min.js +bootstrap-toggle.min.css +d3.v3.min.js +ace.js http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/0be35d62/INSTALL -- diff --git a/INSTALL b/INSTALL new file mode 100644 index 000..7e5bf50 --- /dev/null +++ b/INSTALL @@ -0,0 +1,9 @@ +# INSTALL / BUILD instruction for Apache Airflow (incubating) +# fetch the tarball and untar the source + +# [optional] run Apache RAT (release audit tool) to validate license headers +# RAT docs here: https://creadur.apache.org/rat/ +java -jar apache-rat.jar -E ./.rat-excludes -d . + +# install the release +python setup.py install
svn commit: r20178 - /dev/incubator/airflow/
Author: maximebeauchemin Date: Fri Jun 23 21:08:14 2017 New Revision: 20178 Log: Remove everything but apache-airflow-1.8.2rc2+incubating.tar.gz* Removed: dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.asc dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.md5 dev/incubator/airflow/airflow-1.8.0rc5+apache.incubating.tar.gz.sha dev/incubator/airflow/apache-airflow-1.8.1rc0+apache.incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha dev/incubator/airflow/apache-airflow-1.8.1rc1+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.1rc1+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.1rc1+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.1rc1+incubating.tar.gz.sha dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.1rc2+incubating.tar.gz.sha dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz.sha
svn commit: r20173 - /release/incubator/airflow/KEYS
Author: maximebeauchemin Date: Fri Jun 23 17:54:51 2017 New Revision: 20173 Log: Appending Max's public key to KEYS file Modified: release/incubator/airflow/KEYS Modified: release/incubator/airflow/KEYS == --- release/incubator/airflow/KEYS (original) +++ release/incubator/airflow/KEYS Fri Jun 23 17:54:51 2017 @@ -127,3 +127,42 @@ G6K3WOWXER2Cu5pEJo00Nik7u9qZKi5jHmZPDFNX VI92yOQJ6LdUq0ze/hbHbZs3 =11vL -END PGP PUBLIC KEY BLOCK- + +pub rsa2048 2017-06-13 [SC] [expires: 2019-06-13] + 99E202824A25EAA3750550BEE6F0505CC7BC7E0D +uid [ultimate] Maxime Beauchemin +sig 3E6F0505CC7BC7E0D 2017-06-13 Maxime Beauchemin +sub rsa2048 2017-06-13 [E] [expires: 2019-06-13] +sig E6F0505CC7BC7E0D 2017-06-13 Maxime Beauchemin + +-BEGIN PGP PUBLIC KEY BLOCK- + +mQENBFlAHjwBCAC2+i3yhs3vIBq0OT+7w2/sXmRO6cxIjQRqyRuQBC7BgZbYEr7R +Vni0F5Qv7jOWUXB1VpzQTwlIIfsKvnCAy+6psYJ9qLISTLMOxOTkBLeyEm8ryADd +RKWqH5NEHYlhsoceaEOsPbPWxvmEvHMSBiIyTPSqbzD4fQ8A95gInR+kyz5ZbHkM +vRdcHvGWeMNBnjI7f+dAcGxnFR5sfH63Hj+uTHLNijNu/XFvjE16LLTdITF5u9VY +3b0DDPdH8JdXKLPYV7yNyC8kEqwakFsrEiksg/OvKWHz1QbcLJ9wg64yEtX5A1PX +Ut4iWWsRZ7RNGnz3J+PYbMcRvArDWe1txSpBABEBAAG0L01heGltZSBCZWF1Y2hl +bWluIDxtYXhpbWViZWF1Y2hlbWluQGFwYWNoZS5vcmc+iQFUBBMBCAA+FiEEmeIC +gkol6qN1BVC+5vBQXMe8fg0FAllAHjwCGwMFCQPCZwAFCwkIBwIGFQgJCgsCBBYC +AwECHgECF4AACgkQ5vBQXMe8fg3eMAf/fQNaqg+AFVsjy877rfJhdJ9XSP94gZ9t +qS3/RmnvsMe76Mrydrsp/9t9Yh5NHvm9ZnX5ZEg6cn2T3tFMzVNKzm0ExDYKm5x8 +eRGtO6zgXi7Qlv+NRj3sRr00Nc+ZiVmSwMDApvQBFhJ9l7fnLNUP6Xg0sqv8fqu2 +tPgxP08F1wmzsk+MK0z6kYRDSDppzbDmb1V/zA3hbRqyecUP0Cky2yMn4dKO7Gr8 +tNTxHT3i/Fs5hJjY4XfrVPlstieqW7pUCCrvXOiY3cbXhMNdZCwZScLhE256gp4q +BErXvjZxYAija2x6t025oYkaKWKUVEvbM8j+TNMzS8XzMkKhzf3yVrkBDQRZQB48 +AQgAujmjGvrHRzFXIqmgTq7yUDd/oNJT8RGy4NlgFUE0jZD778UAuCCU/Sr2jzZZ +wV7u79q/yNyLQGjNBtHXsrd9tPJiXFuPziaytG9xhX8rDnsPuUynviM2EvV2Y5t1 ++bAWNi9CgNqMfOA9nV/QF8nXya/Dz1VU74yovn/q/nGBc9c4P4UR4/1j+3M054JY +VgB6n/i0Ly0Xr6XXm2X8e4htyLuLV6YPHdQTyhkgakd63nUfJOoc8CP5axUkDfoj +329/48W6lGiMNvX+gg8/aZpFEFg2BeGrPtBtE98acjqDGartYRd7nTdCSm1qn3vM +6PtgNQ0HtvqSRWHFrmMDkPDLRwARAQABiQE8BBgBCAAmFiEEmeICgkol6qN1BVC+ +5vBQXMe8fg0FAllAHjwCGwwFCQPCZwAACgkQ5vBQXMe8fg1nWAf+P4oW50zhG8UY +gl4l8/nzCbPpFS04ld+KDylpbh03PUNKXA0upRXVDMlgBXS+7vBdMPPNwXKVmdi7 +W6B3B/OPJaa7GJnWscOtwi2b/jhCdWdJaIqivcQTnporhY3q02ogmtcpMDFkl3Pz +oM619PyirqB/LHEFDSgRtVyrGb4KqOEsLRm3wxVmxLnNRSdG8kL9eN0bkwssura3 +roqVcBw8xc7pXYWQhruY98WJG3mSEaqPQ0aEW6V5EamGvDihh+fQM79OQcmJ8j5B +H/htsNmsKjmOgjnW5/acFWoyEfkC2QAafO8x9CV3/WPVlKQx5c430ErVcd9lTslm +60/1YPmBlA== +=G8lD +-END PGP PUBLIC KEY BLOCK-
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.8.2rc2 [created] 302520828
[2/2] incubator-airflow git commit: Updating CHANGELOG for 1.8.2rc2
Updating CHANGELOG for 1.8.2rc2 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/30252082 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/30252082 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/30252082 Branch: refs/heads/v1-8-test Commit: 302520828cdce65a4efa65d495bb0b5d05b35069 Parents: 9a53e66 Author: Maxime Beauchemin Authored: Thu Jun 22 15:26:14 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 22 15:26:14 2017 -0700 -- CHANGELOG.txt | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/30252082/CHANGELOG.txt -- diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 8502267..c4e2d6d 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,6 +1,7 @@ AIRFLOW 1.8.2rc1, 2017-06-05 - +9a53e66 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When Testing Booleans 333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all downstream tasks 93825d5 Re-enable caching for hadoop components 33a9dcb Pin Hive and Hadoop to a specific version and create writable warehouse dir
[1/2] incubator-airflow git commit: [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When Testing Booleans
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test 3975d3dad -> 302520828 [AIRFLOW-809][AIRFLOW-1] Use __eq__ ColumnOperator When Testing Booleans The .is_ ColumnOperator causes the SqlAlchemy's MSSQL dialect to produce IS 0 when given a value of False rather than a value of None. The __eq__ ColumnOperator does this same test with the added benefit that it will modify the resulting expression from and == to a IS NULL when the target is None. This change replaces all is_ ColumnOperators that are doing boolean comparisons and leaves all is_ ColumnOperators that are checking for None values. Closes #2022 from gritlogic/AIRFLOW-809 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9a53e663 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9a53e663 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9a53e663 Branch: refs/heads/v1-8-test Commit: 9a53e66390670c7a9e4206f0a3ef4a19c1baae72 Parents: 3975d3d Author: Chad Henderson Authored: Sun Feb 19 10:03:18 2017 +0100 Committer: Maxime Beauchemin Committed: Thu Jun 22 15:19:45 2017 -0700 -- airflow/models.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9a53e663/airflow/models.py -- diff --git a/airflow/models.py b/airflow/models.py index 737d4c8..4e2abd7 100755 --- a/airflow/models.py +++ b/airflow/models.py @@ -469,7 +469,7 @@ class DagBag(BaseDagBag, LoggingMixin): def paused_dags(self): session = settings.Session() dag_ids = [dp.dag_id for dp in session.query(DagModel).filter( -DagModel.is_paused.is_(True))] +DagModel.is_paused.__eq__(True))] session.commit() session.close() return dag_ids @@ -2858,7 +2858,7 @@ class DAG(BaseDag, LoggingMixin): DR.dag_id == self.dag_id, ) if not include_externally_triggered: -qry = qry.filter(DR.external_trigger.is_(False)) +qry = qry.filter(DR.external_trigger.__eq__(False)) qry = qry.order_by(DR.execution_date.desc())
svn commit: r20167 - in /dev/incubator/airflow: apache-airflow-1.8.2rc2+incubating.tar.gz apache-airflow-1.8.2rc2+incubating.tar.gz.asc apache-airflow-1.8.2rc2+incubating.tar.gz.md5 apache-airflow-1.8
Author: maximebeauchemin Date: Thu Jun 22 22:24:40 2017 New Revision: 20167 Log: Altering 1.8.2rc2 files, last time Modified: dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.sha Modified: dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz == Binary files - no diff available. Modified: dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.asc == --- dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.asc (original) +++ dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.asc Thu Jun 22 22:24:40 2017 @@ -1,11 +1,11 @@ -BEGIN PGP SIGNATURE- -iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAllMPWsACgkQ5vBQXMe8 -fg3+awf+PfClh8eB9DuOMN2Xkkem1LHWuYSDgfI3MCUGvhH8TFB/x+YIqf/HrKyX -m4sJ9GmBuLVtw/nBYXsOQOEPynjf0pf6VOxn0GhLD9TZkbWochGXmV60OBA0xtgK -5JRcsoaBx7TCno4BQC9fk3KwT2TgnNaPBANbH8z0GZUqOrcb/if/rhKQZR2ffTid -MPDFGCcNr/KOGUKANc+xGI4j6ywBk0fji2RUDDGOw7MW8WrQLfdkP/jCEbL0X+N6 -5GulqbuQ+qTjpii4lHgZM1Qv+a0v483qFZ0gfi3xRl1h+MluR4sPtO2c3TMehc6B -UjE/2sisaH3ZvT2lUPXEvC0xXv0fbw== -=HvHI +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAllMQ18ACgkQ5vBQXMe8 +fg2E/Qf/Q4LT2zVaqBxM/2eOL6y8h92953dV+SQsP6pbDkzBjOzxjvS1iAz8Z4yk +agyHB+TvgZVgktrp4HKEL/q0mVZ3JLbebhoQacld0NQUsY7YPaxZYWzfEWsa2Cf4 +wtKTVMokUXixNj+p1+SXu+mu30ptRt9uNG2f8giZTEx+znxArs2zb7W+7g5qRjfU +1P4u5WHEvMgLrskA5f5qWE6PbjFZgYQKMZxaW9+hoRDpigLZglIvdjs7NhUY30gh +KkLNh4FR6C/fp+xmFMxSBW5Ewu/Mtr0DfBxZXNi9ZdLgmLjo9pQzSTDuHy5fpAcj +wUblTAV8hGAOh1sVqVh5fPIQAK2HsQ== +=iwny -END PGP SIGNATURE- Modified: dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.md5 == --- dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.md5 (original) +++ dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.md5 Thu Jun 22 22:24:40 2017 @@ -1 +1,2 @@ -apache-airflow-1.8.2rc2.tar.gz: 7D 78 A7 CF 5A 9C DC 3A E9 CA A7 D7 50 4B 93 14 +apache-airflow-1.8.2rc2+incubating.tar.gz: +08 93 76 31 CA A9 76 C5 07 BD F4 8D B2 2F 1C A3 Modified: dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.sha == --- dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.sha (original) +++ dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.sha Thu Jun 22 22:24:40 2017 @@ -1,4 +1,3 @@ -apache-airflow-1.8.2rc2.tar.gz: 6848454E BE933304 44FE5BBF B25E2FFB 9874 -44991852 B052098A AB3F57A1 2E4012E6 619D57FF -A6AFE7F3 CA834CFA 3B77EC8B 7D402052 06F64D61 -8B9622D1 +apache-airflow-1.8.2rc2+incubating.tar.gz: +DA650C6A 1E42CB59 AE233239 1AC16C01 CD73262A 93F3082D 979C7177 58011B71 DE66C802 + 90E3FBB9 B3C965AF 97DF32A7 CEBF6900 0600921B 8636FE3D 64E0CFDA
svn commit: r20166 - /dev/incubator/airflow/
Author: maximebeauchemin Date: Thu Jun 22 22:16:22 2017 New Revision: 20166 Log: Move 1.8.2rc2 files again Added: dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz.asc - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz.md5 - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc1+incubating.tar.gz.sha - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.sha dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.asc - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.md5 - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2+incubating.tar.gz.sha - copied unchanged from r20165, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.sha Removed: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.sha dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.sha
svn commit: r20165 - /dev/incubator/airflow/
Author: maximebeauchemin Date: Thu Jun 22 22:08:34 2017 New Revision: 20165 Log: Move 1.8.2rc2 files Added: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz - copied unchanged from r20164, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.asc - copied unchanged from r20164, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.md5 - copied unchanged from r20164, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz+incubating.tar.gz.sha - copied unchanged from r20164, dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.sha Removed: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.sha
svn commit: r20164 - in /dev/incubator/airflow: apache-airflow-1.8.2rc2.tar.gz apache-airflow-1.8.2rc2.tar.gz.asc apache-airflow-1.8.2rc2.tar.gz.md5 apache-airflow-1.8.2rc2.tar.gz.sha
Author: maximebeauchemin Date: Thu Jun 22 22:05:58 2017 New Revision: 20164 Log: Add 1.8.2rc2 files Added: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz (with props) dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.sha Added: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz == Binary file - no diff available. Propchange: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.asc == --- dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.asc (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.asc Thu Jun 22 22:05:58 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAllMPWsACgkQ5vBQXMe8 +fg3+awf+PfClh8eB9DuOMN2Xkkem1LHWuYSDgfI3MCUGvhH8TFB/x+YIqf/HrKyX +m4sJ9GmBuLVtw/nBYXsOQOEPynjf0pf6VOxn0GhLD9TZkbWochGXmV60OBA0xtgK +5JRcsoaBx7TCno4BQC9fk3KwT2TgnNaPBANbH8z0GZUqOrcb/if/rhKQZR2ffTid +MPDFGCcNr/KOGUKANc+xGI4j6ywBk0fji2RUDDGOw7MW8WrQLfdkP/jCEbL0X+N6 +5GulqbuQ+qTjpii4lHgZM1Qv+a0v483qFZ0gfi3xRl1h+MluR4sPtO2c3TMehc6B +UjE/2sisaH3ZvT2lUPXEvC0xXv0fbw== +=HvHI +-END PGP SIGNATURE- Added: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.md5 == --- dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.md5 (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.md5 Thu Jun 22 22:05:58 2017 @@ -0,0 +1 @@ +apache-airflow-1.8.2rc2.tar.gz: 7D 78 A7 CF 5A 9C DC 3A E9 CA A7 D7 50 4B 93 14 Added: dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.sha == --- dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.sha (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc2.tar.gz.sha Thu Jun 22 22:05:58 2017 @@ -0,0 +1,4 @@ +apache-airflow-1.8.2rc2.tar.gz: 6848454E BE933304 44FE5BBF B25E2FFB 9874 +44991852 B052098A AB3F57A1 2E4012E6 619D57FF +A6AFE7F3 CA834CFA 3B77EC8B 7D402052 06F64D61 +8B9622D1
[2/2] incubator-airflow git commit: Bump to 1.8.2rc2
Bump to 1.8.2rc2 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3975d3da Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3975d3da Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3975d3da Branch: refs/heads/v1-8-test Commit: 3975d3dadc35a7f1606976edb21c145354905993 Parents: f58cfa3 Author: Maxime Beauchemin Authored: Thu Jun 22 14:52:52 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 22 14:52:52 2017 -0700 -- airflow/version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3975d3da/airflow/version.py -- diff --git a/airflow/version.py b/airflow/version.py index 5b7dce4..ce57c88 100644 --- a/airflow/version.py +++ b/airflow/version.py @@ -13,4 +13,4 @@ # limitations under the License. # -version = '1.8.2rc1' +version = '1.8.2rc2'
[1/2] incubator-airflow git commit: Update CHANGELOG for 1.8.2rc
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test 333e0b3eb -> 3975d3dad Update CHANGELOG for 1.8.2rc Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f58cfa3e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f58cfa3e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f58cfa3e Branch: refs/heads/v1-8-test Commit: f58cfa3e1a98b613625b3f9c3e91cb75ba4e85ca Parents: 333e0b3 Author: Maxime Beauchemin Authored: Wed Jun 21 15:14:12 2017 -0700 Committer: Maxime Beauchemin Committed: Wed Jun 21 15:14:12 2017 -0700 -- CHANGELOG.txt | 5 + 1 file changed, 5 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f58cfa3e/CHANGELOG.txt -- diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 3c60366..8502267 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,6 +1,11 @@ AIRFLOW 1.8.2rc1, 2017-06-05 - +333e0b3 [AIRFLOW-1296] Propagate SKIPPED to all downstream tasks +93825d5 Re-enable caching for hadoop components +33a9dcb Pin Hive and Hadoop to a specific version and create writable warehouse dir +7cff6cd [AIRFLOW-1308] Disable nanny usage for Dask +c6a09c4 Updating CHANGELOG for 1.8.2rc1 570b2ed [AIRFLOW-1294] Backfills can loose tasks to execute 3f48d48 [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements e10af9a [AIRFLOW-XXX] Set version to 1.8.2rc1
[1/2] incubator-airflow git commit: [AIRFLOW-1339] Add Drivy to the list of users
Repository: incubator-airflow Updated Branches: refs/heads/master 9958aa9d5 -> e0491bcb2 [AIRFLOW-1339] Add Drivy to the list of users Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/ea4ee03a Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/ea4ee03a Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/ea4ee03a Branch: refs/heads/master Commit: ea4ee03af0cea8ecdc8f9bddf3a6513dcd7e539f Parents: 9958aa9 Author: Antoine Augusti Authored: Thu Jun 22 15:38:15 2017 +0200 Committer: GitHub Committed: Thu Jun 22 15:38:15 2017 +0200 -- README.md | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/ea4ee03a/README.md -- diff --git a/README.md b/README.md index c08b98b..49280b4 100644 --- a/README.md +++ b/README.md @@ -179,6 +179,7 @@ Currently **officially** using Airflow: 1. [Zendesk](https://www.github.com/zendesk) 1. [Zenly](https://zen.ly) [[@cerisier](https://github.com/cerisier) & [@jbdalido](https://github.com/jbdalido)] 1. [99](https://99taxis.com) [[@fbenevides](https://github.com/fbenevides), [@gustavoamigo](https://github.com/gustavoamigo) & [@mmmaia](https://github.com/mmmaia)] +1. [Drivy](https://www.drivy.com) [[@AntoineAugusti](https://github.com/AntoineAugusti)] ## Links
[2/2] incubator-airflow git commit: Merge pull request #2389 from AntoineAugusti/drivy
Merge pull request #2389 from AntoineAugusti/drivy Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e0491bcb Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e0491bcb Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e0491bcb Branch: refs/heads/master Commit: e0491bcb2057dd85bf2dd80266ae4ff5b1b7eaed Parents: 9958aa9 ea4ee03 Author: Maxime Beauchemin Authored: Thu Jun 22 09:26:41 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 22 09:26:41 2017 -0700 -- README.md | 1 + 1 file changed, 1 insertion(+) --
[1/2] incubator-airflow git commit: [AIRFLOW-1317] Fix minor issues in API reference
Repository: incubator-airflow Updated Branches: refs/heads/master d8891d906 -> 7b620391a [AIRFLOW-1317] Fix minor issues in API reference Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/1ae7e5b9 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/1ae7e5b9 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/1ae7e5b9 Branch: refs/heads/master Commit: 1ae7e5b9af8ec2fdb40bcd4b9587afbd7861488e Parents: d8891d9 Author: Kengo Seki Authored: Sun Jun 18 04:26:57 2017 -0400 Committer: Kengo Seki Committed: Tue Jun 20 20:16:35 2017 -0400 -- airflow/macros/__init__.py | 1 - airflow/models.py| 10 ++ airflow/operators/sensors.py | 4 +++- docs/code.rst| 8 4 files changed, 13 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1ae7e5b9/airflow/macros/__init__.py -- diff --git a/airflow/macros/__init__.py b/airflow/macros/__init__.py index 59b9a25..6f80c04 100644 --- a/airflow/macros/__init__.py +++ b/airflow/macros/__init__.py @@ -13,7 +13,6 @@ # limitations under the License. from __future__ import absolute_import -from random import random from datetime import datetime, timedelta import dateutil import time http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1ae7e5b9/airflow/models.py -- diff --git a/airflow/models.py b/airflow/models.py index 30e18a4..c628958 100755 --- a/airflow/models.py +++ b/airflow/models.py @@ -910,7 +910,7 @@ class TaskInstance(Base): :param local: Whether to run the task locally :type local: bool :param pickle_id: If the DAG was serialized to the DB, the ID -associated with the pickled DAG +associated with the pickled DAG :type pickle_id: unicode :param file_path: path to the file containing the DAG definition :param raw: raw mode (needs more details) @@ -1239,6 +1239,7 @@ class TaskInstance(Base): def get_dagrun(self, session): """ Returns the DagRun for this TaskInstance + :param session: :return: DagRun """ @@ -2684,7 +2685,7 @@ class DAG(BaseDag, LoggingMixin): :param orientation: Specify DAG orientation in graph view (LR, TB, RL, BT) :type orientation: string :param catchup: Perform scheduler catchup (or only run latest)? Defaults to True -"type catchup: bool" +:type catchup: bool """ def __init__( @@ -2989,6 +2990,7 @@ class DAG(BaseDag, LoggingMixin): """ Returns the dag run for a given execution date if it exists, otherwise none. + :param execution_date: The execution date of the DagRun to find. :param session: :return: The DagRun if found, otherwise None. @@ -3093,6 +3095,7 @@ class DAG(BaseDag, LoggingMixin): Heavily inspired by: http://blog.jupo.org/2012/04/06/topological-sorting-acyclic-directed-graphs/ + :return: list of tasks in topological order """ @@ -3479,7 +3482,6 @@ class DAG(BaseDag, LoggingMixin): :param dag: the DAG object to save to the DB :type dag: DAG -:own :param sync_time: The time that the DAG should be marked as sync'ed :type sync_time: datetime :return: None @@ -3528,7 +3530,7 @@ class DAG(BaseDag, LoggingMixin): the expiration date. These DAGs were likely deleted. :param expiration_date: set inactive DAGs that were touched before this -time +time :type expiration_date: datetime :return: None """ http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1ae7e5b9/airflow/operators/sensors.py -- diff --git a/airflow/operators/sensors.py b/airflow/operators/sensors.py index c28e88b..4f276ad 100644 --- a/airflow/operators/sensors.py +++ b/airflow/operators/sensors.py @@ -405,6 +405,7 @@ class HdfsSensor(BaseSensorOperator): def filter_for_filesize(result, size=None): """ Will test the filepath result and test if its size is at least self.filesize + :param result: a list of dicts returned by Snakebite ls :param size: the file size in MB a file should be at least to trigger True :return: (bool) depending on the matching criteria @@ -420,10 +421,11 @@ class HdfsSensor(BaseSensorOperator): def filter_for_ignored_ext(result, ignored_ext, ignore_copying): """ Will filter if instructed to do so the result to remove matching criteria +
[2/2] incubator-airflow git commit: Merge pull request #2374 from sekikn/AIRFLOW-1317
Merge pull request #2374 from sekikn/AIRFLOW-1317 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7b620391 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7b620391 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7b620391 Branch: refs/heads/master Commit: 7b620391a4e71fd19dff037a859dd39f132edb8c Parents: d8891d9 1ae7e5b Author: Maxime Beauchemin Authored: Tue Jun 20 22:34:03 2017 -0700 Committer: Maxime Beauchemin Committed: Tue Jun 20 22:34:03 2017 -0700 -- airflow/macros/__init__.py | 1 - airflow/models.py| 10 ++ airflow/operators/sensors.py | 4 +++- docs/code.rst| 8 4 files changed, 13 insertions(+), 10 deletions(-) --
svn commit: r20024 - in /dev/incubator/airflow: apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.m
Author: maximebeauchemin Date: Tue Jun 13 18:08:42 2017 New Revision: 20024 Log: Add 1.8.2rc1 files Added: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz (with props) dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.md5 dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.sha Added: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz == Binary file - no diff available. Propchange: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz -- svn:mime-type = application/octet-stream Added: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc == --- dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.asc Tue Jun 13 18:08:42 2017 @@ -0,0 +1,11 @@ +-BEGIN PGP SIGNATURE- + +iQEzBAABCAAdFiEEmeICgkol6qN1BVC+5vBQXMe8fg0FAllAKR4ACgkQ5vBQXMe8 +fg3kZQf+MPxSbjWxkfF15MID5Ru2ArVd6WMYlfr9pTSRpP3H3w1863hspcgtebbf +3+C5jVtKTyeCTSmqk/60hAqpal3awYCMjHpUKB55OsDTE9oHc0hhUG9C7GSLY7Ey +7Mk9EqcdSiIiWqldzMsCXkRCvVZ5+72psIPUGxql388uh7Xbuf245Fg1QL98y1dX +VIwGuFEyNippLXrVR4GJcdoXrlQfEYk9g/NXIUrNmnagnYS4/AlqYjaAGQNv7zLq +R05+jn8Pg0o7KIMJgOLogjz7WY7e1lTW97LnptPv4lvBr9Yc5P8pPLY3kJYeflRp +dWDKht/EuEyrpeXbBTYpicmTDi0EYw== +=UKky +-END PGP SIGNATURE- Added: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.md5 == --- dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.md5 (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.md5 Tue Jun 13 18:08:42 2017 @@ -0,0 +1,2 @@ +apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz: +4B F8 B5 7A 54 79 18 04 9F 93 08 89 42 54 0E 2E Added: dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.sha == --- dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.sha (added) +++ dev/incubator/airflow/apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz.sha Tue Jun 13 18:08:42 2017 @@ -0,0 +1,3 @@ +apache-airflow-1.8.2rc1.tar.gz+incubating.tar.gz: +07372F4C 0649A3DE E5B38100 8EF274C0 0BBD2C9A 3BBDED63 8AFAA7A9 0C0EA054 D8816EDB + A561155C D0AB9E37 E0CBEBB9 924C62AB 8FCD0116 8E238A16 D2BC9E3D
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.8.2rc1 [created] c6a09c47e
incubator-airflow git commit: Updating CHANGELOG for 1.8.2rc1
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test 570b2ed3e -> c6a09c47e Updating CHANGELOG for 1.8.2rc1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c6a09c47 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c6a09c47 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c6a09c47 Branch: refs/heads/v1-8-test Commit: c6a09c47e6402a05d2477e6431c6d717e2b5a3ba Parents: 570b2ed Author: Maxime Beauchemin Authored: Tue Jun 13 08:25:28 2017 -0700 Committer: Maxime Beauchemin Committed: Tue Jun 13 08:25:28 2017 -0700 -- CHANGELOG.txt | 2 ++ 1 file changed, 2 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c6a09c47/CHANGELOG.txt -- diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 8d59fa1..3c60366 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,6 +1,8 @@ AIRFLOW 1.8.2rc1, 2017-06-05 - +570b2ed [AIRFLOW-1294] Backfills can loose tasks to execute +3f48d48 [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements e10af9a [AIRFLOW-XXX] Set version to 1.8.2rc1 69bd269 [AIRFLOW-1160] Update Spark parameters for Mesos 9692510 [AIRFLOW 1149][AIRFLOW-1149] Allow for custom filters in Jinja2 templates
[1/2] incubator-airflow git commit: [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test a2dd2465a -> 570b2ed3e [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements JIRA: https://issues.apache.org/jira/browse/AIRFLOW-1291 * Update NOTICE with proper year range for ASF copyright * Break down LICENSE into licenses/LICENSE-[project].txt * add license header to jqClock.min.js [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements * Update NOTICE with proper year range for ASF copyright * Break down LICENSE into licenses/LICENSE-[project].txt * add license header to jqClock.min.js fix license check Closes #2354 from mistercrunch/copyright_license_touchups Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/3f48d48e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/3f48d48e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/3f48d48e Branch: refs/heads/v1-8-test Commit: 3f48d48e1b76f7db9db0db11b4f247d10f7e4006 Parents: a2dd246 Author: Maxime Beauchemin Authored: Mon Jun 12 10:47:56 2017 -0700 Committer: Maxime Beauchemin Committed: Mon Jun 12 10:50:27 2017 -0700 -- .rat-excludes | 2 +- LICENSE | 378 +++ NOTICE | 3 +- airflow/www/static/jqClock.min.js | 7 + licenses/LICENSE-ace.txt| 24 ++ licenses/LICENSE-bootstrap.txt | 48 licenses/LICENSE-d3-tip.txt | 17 ++ licenses/LICENSE-d3js.txt | 30 +++ licenses/LICENSE-dagre-d3.txt | 21 ++ licenses/LICENSE-datatables.txt | 17 ++ licenses/LICENSE-flask-kerberos.txt | 23 ++ licenses/LICENSE-jquery.txt | 69 ++ licenses/LICENSE-jsclockplugin.txt | 7 + licenses/LICENSE-underscorejs.txt | 23 ++ licenses/LICENSE-webgl-2d.txt | 23 ++ 15 files changed, 348 insertions(+), 344 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3f48d48e/.rat-excludes -- diff --git a/.rat-excludes b/.rat-excludes index 1238abb..d5b8b4e 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -24,4 +24,4 @@ CHANGELOG.txt # it is compatible according to http://www.apache.org/legal/resolved.html#category-a kerberos_auth.py airflow_api_auth_backend_kerberos_auth_py.html - +licenses/* http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/3f48d48e/LICENSE -- diff --git a/LICENSE b/LICENSE index a70a898..19ba93f 100644 --- a/LICENSE +++ b/LICENSE @@ -200,345 +200,39 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -This product includes jQuery (http://jquery.org - MIT license), Copyright © 2016, John Resig. -/* -Copyright JS Foundation and other contributors, https://js.foundation/ - -This software consists of voluntary contributions made by many -individuals. For exact contribution history, see the revision history -available at https://github.com/jquery/jquery - -The following license applies to all parts of this software except as -documented below: - - - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - - -All files located in the node_modules and external directories are -externally maintained libraries used by this software which have their -own licenses; we recommend you read them, as their terms may differ from -the terms above. -*/ - -This product includes Parallel Coordinates (https://syntagmatic.github.io/parallel-coordinates - BSD License), Copyright (c) 2012, Kai Chang. -For airflow/www/static/para/parallel.js: -
[2/2] incubator-airflow git commit: [AIRFLOW-1294] Backfills can loose tasks to execute
[AIRFLOW-1294] Backfills can loose tasks to execute
In backfills we can loose tasks to execute due to
a task
setting its own state to NONE if concurrency
limits are reached,
this makes them fall outside of the scope the
backfill is
managing hence they will not be executed.
Dear Airflow maintainers,
Please accept this PR. I understand that it will
not be reviewed until I have checked off all the
steps below!
### JIRA
- [X] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "[AIRFLOW-XXX] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-1294
### Description
- [X] Here are some details about my PR, including
screenshots of any UI changes:
In backfills we can loose tasks to execute due to
a task
setting its own state to NONE if concurrency
limits are reached,
this makes them fall outside of the scope the
backfill is
managing hence they will not be executed.
### Tests
- [X] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
Should be covered by current tests, will adjust if
required.
### Commits
- [X] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
mistercrunch aoen saguziel This is a simplified
fix that should be easier to digest in 1.8.2. It
does not address all underlying issues as in
https://github.com/apache/incubator-
airflow/pull/2356 , but those can be addressed
separately and in smaller bits.
Closes #2360 from bolkedebruin/fix_race_backfill_2
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/570b2ed3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/570b2ed3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/570b2ed3
Branch: refs/heads/v1-8-test
Commit: 570b2ed3ef01123dace11b620b4fcafde3bcd8b8
Parents: 3f48d48
Author: Bolke de Bruin
Authored: Tue Jun 13 08:21:26 2017 -0700
Committer: Maxime Beauchemin
Committed: Tue Jun 13 08:23:33 2017 -0700
--
airflow/jobs.py | 47 -
tests/jobs.py | 85
2 files changed, 118 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/570b2ed3/airflow/jobs.py
--
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 457966f..c517350 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1668,7 +1668,8 @@ class BackfillJob(BaseJob):
def _update_counters(self, started, succeeded, skipped, failed,
tasks_to_run):
"""
-Updates the counters per state of the tasks that were running
+Updates the counters per state of the tasks that were running. Can
re-add
+to tasks to run in case required.
:param started:
:param succeeded:
:param skipped:
@@ -1700,6 +1701,20 @@ class BackfillJob(BaseJob):
.format(ti))
started.pop(key)
tasks_to_run[key] = ti
+# special case: The state of the task can be set to NONE by the
task itself
+# when it reaches concurrency limits. It could also happen when
the state
+# is changed externally, e.g. by clearing tasks from the ui. We
need to cover
+# for that as otherwise those tasks would fall outside of the
scope of
+# the backfill suddenly.
+elif ti.state == State.NONE:
+self.logger.warning("FIXME: task instance {} state was set to "
+"None externally or reaching concurrency
limits. "
+"Re-adding task to queue.".format(ti))
+session = settings.Session()
+ti.set_state(State.SCHEDULED, session=session)
+session.close()
+started.pop(key)
+tasks_to_run[key] = ti
def _manage_executor_state(self, started):
"""
@@ -1909,19 +1924,23 @@ class BackfillJob(BaseJob):
verbose=True):
ti.refresh_from_db(lock_for_update=True,
session=session)
if ti.state == State.SCHEDULED or ti.stat
incubator-airflow git commit: [AIRFLOW-1294] Backfills can loose tasks to execute
Repository: incubator-airflow
Updated Branches:
refs/heads/master c2b80e928 -> 6e3bcd318
[AIRFLOW-1294] Backfills can loose tasks to execute
In backfills we can loose tasks to execute due to
a task
setting its own state to NONE if concurrency
limits are reached,
this makes them fall outside of the scope the
backfill is
managing hence they will not be executed.
Dear Airflow maintainers,
Please accept this PR. I understand that it will
not be reviewed until I have checked off all the
steps below!
### JIRA
- [X] My PR addresses the following [Airflow JIRA]
(https://issues.apache.org/jira/browse/AIRFLOW/)
issues and references them in the PR title. For
example, "[AIRFLOW-XXX] My Airflow PR"
-
https://issues.apache.org/jira/browse/AIRFLOW-1294
### Description
- [X] Here are some details about my PR, including
screenshots of any UI changes:
In backfills we can loose tasks to execute due to
a task
setting its own state to NONE if concurrency
limits are reached,
this makes them fall outside of the scope the
backfill is
managing hence they will not be executed.
### Tests
- [X] My PR adds the following unit tests __OR__
does not need testing for this extremely good
reason:
Should be covered by current tests, will adjust if
required.
### Commits
- [X] My commits all reference JIRA issues in
their subject lines, and I have squashed multiple
commits if they address the same issue. In
addition, my commits follow the guidelines from
"[How to write a good git commit
message](http://chris.beams.io/posts/git-
commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not
"adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
mistercrunch aoen saguziel This is a simplified
fix that should be easier to digest in 1.8.2. It
does not address all underlying issues as in
https://github.com/apache/incubator-
airflow/pull/2356 , but those can be addressed
separately and in smaller bits.
Closes #2360 from bolkedebruin/fix_race_backfill_2
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6e3bcd31
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6e3bcd31
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6e3bcd31
Branch: refs/heads/master
Commit: 6e3bcd31868ac834e1d8a9305e66ee1b8c586eb1
Parents: c2b80e9
Author: Bolke de Bruin
Authored: Tue Jun 13 08:21:26 2017 -0700
Committer: Maxime Beauchemin
Committed: Tue Jun 13 08:21:26 2017 -0700
--
airflow/jobs.py | 47 -
tests/jobs.py | 85
2 files changed, 118 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6e3bcd31/airflow/jobs.py
--
diff --git a/airflow/jobs.py b/airflow/jobs.py
index 57688d3..2b4350e 100644
--- a/airflow/jobs.py
+++ b/airflow/jobs.py
@@ -1669,7 +1669,8 @@ class BackfillJob(BaseJob):
def _update_counters(self, started, succeeded, skipped, failed,
tasks_to_run):
"""
-Updates the counters per state of the tasks that were running
+Updates the counters per state of the tasks that were running. Can
re-add
+to tasks to run in case required.
:param started:
:param succeeded:
:param skipped:
@@ -1701,6 +1702,20 @@ class BackfillJob(BaseJob):
.format(ti))
started.pop(key)
tasks_to_run[key] = ti
+# special case: The state of the task can be set to NONE by the
task itself
+# when it reaches concurrency limits. It could also happen when
the state
+# is changed externally, e.g. by clearing tasks from the ui. We
need to cover
+# for that as otherwise those tasks would fall outside of the
scope of
+# the backfill suddenly.
+elif ti.state == State.NONE:
+self.logger.warning("FIXME: task instance {} state was set to "
+"None externally or reaching concurrency
limits. "
+"Re-adding task to queue.".format(ti))
+session = settings.Session()
+ti.set_state(State.SCHEDULED, session=session)
+session.close()
+started.pop(key)
+tasks_to_run[key] = ti
def _manage_executor_state(self, started):
"""
@@ -1910,19 +1925,23 @@ class BackfillJob(BaseJob):
verbose=True):
ti.refresh_from_db(lock_for_update=
incubator-airflow git commit: [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements
Repository: incubator-airflow Updated Branches: refs/heads/master 355fc92b0 -> 6c55a2219 [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements JIRA: https://issues.apache.org/jira/browse/AIRFLOW-1291 * Update NOTICE with proper year range for ASF copyright * Break down LICENSE into licenses/LICENSE-[project].txt * add license header to jqClock.min.js [AIRFLOW-1291] Update NOTICE and LICENSE files to match ASF requirements * Update NOTICE with proper year range for ASF copyright * Break down LICENSE into licenses/LICENSE-[project].txt * add license header to jqClock.min.js fix license check Closes #2354 from mistercrunch/copyright_license_touchups Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6c55a221 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6c55a221 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6c55a221 Branch: refs/heads/master Commit: 6c55a2219c60e60d7f9a07d80c28e2851b93368f Parents: 355fc92 Author: Maxime Beauchemin Authored: Mon Jun 12 10:47:56 2017 -0700 Committer: Maxime Beauchemin Committed: Mon Jun 12 10:47:56 2017 -0700 -- .rat-excludes | 2 +- LICENSE | 378 +++ NOTICE | 3 +- airflow/www/static/jqClock.min.js | 7 + licenses/LICENSE-ace.txt| 24 ++ licenses/LICENSE-bootstrap.txt | 48 licenses/LICENSE-d3-tip.txt | 17 ++ licenses/LICENSE-d3js.txt | 30 +++ licenses/LICENSE-dagre-d3.txt | 21 ++ licenses/LICENSE-datatables.txt | 17 ++ licenses/LICENSE-flask-kerberos.txt | 23 ++ licenses/LICENSE-jquery.txt | 69 ++ licenses/LICENSE-jsclockplugin.txt | 7 + licenses/LICENSE-underscorejs.txt | 23 ++ licenses/LICENSE-webgl-2d.txt | 23 ++ 15 files changed, 348 insertions(+), 344 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6c55a221/.rat-excludes -- diff --git a/.rat-excludes b/.rat-excludes index 1238abb..d5b8b4e 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -24,4 +24,4 @@ CHANGELOG.txt # it is compatible according to http://www.apache.org/legal/resolved.html#category-a kerberos_auth.py airflow_api_auth_backend_kerberos_auth_py.html - +licenses/* http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6c55a221/LICENSE -- diff --git a/LICENSE b/LICENSE index a70a898..19ba93f 100644 --- a/LICENSE +++ b/LICENSE @@ -200,345 +200,39 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -This product includes jQuery (http://jquery.org - MIT license), Copyright © 2016, John Resig. -/* -Copyright JS Foundation and other contributors, https://js.foundation/ - -This software consists of voluntary contributions made by many -individuals. For exact contribution history, see the revision history -available at https://github.com/jquery/jquery - -The following license applies to all parts of this software except as -documented below: - - - -Permission is hereby granted, free of charge, to any person obtaining -a copy of this software and associated documentation files (the -"Software"), to deal in the Software without restriction, including -without limitation the rights to use, copy, modify, merge, publish, -distribute, sublicense, and/or sell copies of the Software, and to -permit persons to whom the Software is furnished to do so, subject to -the following conditions: - -The above copyright notice and this permission notice shall be -included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, -EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF -MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND -NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE -LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION -OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION -WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. - - - -All files located in the node_modules and external directories are -externally maintained libraries used by this software which have their -own licenses; we recommend you read them, as their terms may differ from -the terms above. -*/ - -This product includes Parallel Coordinates (https://syntagmatic.github.io/parallel-coordinates - BSD License), Copyright (c) 2012, Kai Chang. -For airflow/www/static/para/parallel.js: -/* -Co
[14/18] incubator-airflow git commit: [AIRFLOW-1089] Add Spark application arguments
[AIRFLOW-1089] Add Spark application arguments
Allows arguments to be passed to the Spark
application being
submitted. For example:
- spark-submit --class foo.Bar foobar.jar arg1
arg2
- spark-submit app.py arg1 arg2
Closes #2229 from camshrun/sparkSubmitAppArgs
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/34f072a1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/34f072a1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/34f072a1
Branch: refs/heads/v1-8-test
Commit: 34f072a1716350ec39464d73282127d08b83582c
Parents: 156e90b
Author: Stephan Werges
Authored: Tue Apr 25 11:28:31 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/contrib/hooks/spark_submit_hook.py | 9 +
airflow/contrib/operators/spark_submit_operator.py | 5 +
tests/contrib/hooks/spark_submit_hook.py | 16 +++-
tests/contrib/operators/spark_submit_operator.py | 7 ++-
4 files changed, 35 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/34f072a1/airflow/contrib/hooks/spark_submit_hook.py
--
diff --git a/airflow/contrib/hooks/spark_submit_hook.py
b/airflow/contrib/hooks/spark_submit_hook.py
index 59d28b5..e4ce797 100644
--- a/airflow/contrib/hooks/spark_submit_hook.py
+++ b/airflow/contrib/hooks/spark_submit_hook.py
@@ -56,6 +56,8 @@ class SparkSubmitHook(BaseHook):
:type name: str
:param num_executors: Number of executors to launch
:type num_executors: int
+:param application_args: Arguments for the application being submitted
+:type application_args: list
:param verbose: Whether to pass the verbose flag to spark-submit process
for debugging
:type verbose: bool
"""
@@ -74,6 +76,7 @@ class SparkSubmitHook(BaseHook):
principal=None,
name='default-name',
num_executors=None,
+ application_args=None,
verbose=False):
self._conf = conf
self._conn_id = conn_id
@@ -88,6 +91,7 @@ class SparkSubmitHook(BaseHook):
self._principal = principal
self._name = name
self._num_executors = num_executors
+self._application_args = application_args
self._verbose = verbose
self._sp = None
self._yarn_application_id = None
@@ -183,6 +187,11 @@ class SparkSubmitHook(BaseHook):
# The actual script to execute
connection_cmd += [application]
+# Append any application arguments
+if self._application_args:
+for arg in self._application_args:
+connection_cmd += [arg]
+
logging.debug("Spark-Submit cmd: {}".format(connection_cmd))
return connection_cmd
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/34f072a1/airflow/contrib/operators/spark_submit_operator.py
--
diff --git a/airflow/contrib/operators/spark_submit_operator.py
b/airflow/contrib/operators/spark_submit_operator.py
index f62c395..2a7e3cf 100644
--- a/airflow/contrib/operators/spark_submit_operator.py
+++ b/airflow/contrib/operators/spark_submit_operator.py
@@ -56,6 +56,8 @@ class SparkSubmitOperator(BaseOperator):
:type name: str
:param num_executors: Number of executors to launch
:type num_executors: int
+:param application_args: Arguments for the application being submitted
+:type application_args: list
:param verbose: Whether to pass the verbose flag to spark-submit process
for debugging
:type verbose: bool
"""
@@ -76,6 +78,7 @@ class SparkSubmitOperator(BaseOperator):
principal=None,
name='airflow-spark',
num_executors=None,
+ application_args=None,
verbose=False,
*args,
**kwargs):
@@ -93,6 +96,7 @@ class SparkSubmitOperator(BaseOperator):
self._principal = principal
self._name = name
self._num_executors = num_executors
+self._application_args = application_args
self._verbose = verbose
self._hook = None
self._conn_id = conn_id
@@ -115,6 +119,7 @@ class SparkSubmitOperator(BaseOperator):
principal=self._principal,
name=self._name,
num_executors=self._num_executors,
+application_args=self._application_args,
verbose=self._verbose
)
self._hook.submit(self._application)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/34f072a1/tests/contrib/hooks/spark_submi
[18/18] incubator-airflow git commit: [AIRFLOW 1149][AIRFLOW-1149] Allow for custom filters in Jinja2 templates
[AIRFLOW 1149][AIRFLOW-1149] Allow for custom filters in Jinja2 templates
Closes #2258 from
NielsZeilemaker/jinja_custom_filters
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9744b1a8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9744b1a8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9744b1a8
Branch: refs/heads/v1-8-test
Commit: 9744b1a8d8223fb713156eacbcaf07456d6d43c3
Parents: 07ebc46
Author: Niels Zeilemaker
Authored: Sat Apr 29 17:14:40 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/models.py | 19 ++---
docs/tutorial.rst | 10 +
tests/models.py | 55 ++
3 files changed, 81 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9744b1a8/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 8ef9748..01f8de3 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -2214,11 +2214,13 @@ class BaseOperator(object):
memo[id(self)] = result
for k, v in list(self.__dict__.items()):
-if k not in ('user_defined_macros', 'params'):
+if k not in ('user_defined_macros', 'user_defined_filters',
'params'):
setattr(result, k, copy.deepcopy(v, memo))
result.params = self.params
if hasattr(self, 'user_defined_macros'):
result.user_defined_macros = self.user_defined_macros
+if hasattr(self, 'user_defined_filters'):
+result.user_defined_filters = self.user_defined_filters
return result
def render_template_from_field(self, attr, content, context, jinja_env):
@@ -2609,6 +2611,12 @@ class DAG(BaseDag, LoggingMixin):
templates related to this DAG. Note that you can pass any
type of object here.
:type user_defined_macros: dict
+:param user_defined_filters: a dictionary of filters that will be exposed
+in your jinja templates. For example, passing
+``dict(hello=lambda name: 'Hello %s' % name)`` to this argument allows
+you to ``{{ 'world' | hello }}`` in all jinja templates related to
+this DAG.
+:type user_defined_filters: dict
:param default_args: A dictionary of default parameters to be used
as constructor keyword parameters when initialising operators.
Note that operators have the same hook, and precede those defined
@@ -2647,6 +2655,7 @@ class DAG(BaseDag, LoggingMixin):
full_filepath=None,
template_searchpath=None,
user_defined_macros=None,
+user_defined_filters=None,
default_args=None,
concurrency=configuration.getint('core', 'dag_concurrency'),
max_active_runs=configuration.getint(
@@ -2658,6 +2667,7 @@ class DAG(BaseDag, LoggingMixin):
params=None):
self.user_defined_macros = user_defined_macros
+self.user_defined_filters = user_defined_filters
self.default_args = default_args or {}
self.params = params or {}
@@ -2994,7 +3004,7 @@ class DAG(BaseDag, LoggingMixin):
def get_template_env(self):
"""
Returns a jinja2 Environment while taking into account the DAGs
-template_searchpath and user_defined_macros
+template_searchpath, user_defined_macros and user_defined_filters
"""
searchpath = [self.folder]
if self.template_searchpath:
@@ -3006,6 +3016,8 @@ class DAG(BaseDag, LoggingMixin):
cache_size=0)
if self.user_defined_macros:
env.globals.update(self.user_defined_macros)
+if self.user_defined_filters:
+env.filters.update(self.user_defined_filters)
return env
@@ -3172,10 +3184,11 @@ class DAG(BaseDag, LoggingMixin):
result = cls.__new__(cls)
memo[id(self)] = result
for k, v in list(self.__dict__.items()):
-if k not in ('user_defined_macros', 'params'):
+if k not in ('user_defined_macros', 'user_defined_filters',
'params'):
setattr(result, k, copy.deepcopy(v, memo))
result.user_defined_macros = self.user_defined_macros
+result.user_defined_filters = self.user_defined_filters
result.params = self.params
return result
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9744b1a8/docs/tutorial.rst
--
diff --git a/docs/tutorial.rst b/docs/tutorial.rst
index 97bbe11..ed70823 100644
--- a/docs/tutorial.rst
+++ b/docs/tutorial.rst
@@ -231,6 +2
[11/18] incubator-airflow git commit: [AIRFLOW-1036] Randomize exponential backoff
[AIRFLOW-1036] Randomize exponential backoff
This prevents the thundering herd problem. Using a
combination of
dag_run, task_id, and execution_date makes this
random with respect to
task instances, while still being deterministic
across machines. The
retry delay is within a range that doubles in
size.
Closes #2262 from saguziel/aguziel-random-
exponential-backoff
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/258baf0f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/258baf0f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/258baf0f
Branch: refs/heads/v1-8-test
Commit: 258baf0ff0af9529388a464174dc703d0ad48f5b
Parents: 6503995
Author: Alex Guziel
Authored: Sat Apr 29 17:11:58 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/models.py | 8 +++-
tests/models.py | 15 +++
2 files changed, 18 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/258baf0f/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index bf50c4c..8ef9748 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -1163,13 +1163,19 @@ class TaskInstance(Base):
"""
delay = self.task.retry_delay
if self.task.retry_exponential_backoff:
+min_backoff = int(delay.total_seconds() * (2 ** (self.try_number -
2)))
+# deterministic per task instance
+hash = int(hashlib.sha1("{}#{}#{}#{}".format(self.dag_id,
self.task_id,
+self.execution_date,
self.try_number).encode('utf-8')).hexdigest(), 16)
+# between 0.5 * delay * (2^retry_number) and 1.0 * delay *
(2^retry_number)
+modded_hash = min_backoff + hash % min_backoff
# timedelta has a maximum representable value. The exponentiation
# here means this value can be exceeded after a certain number
# of tries (around 50 if the initial delay is 1s, even fewer if
# the delay is larger). Cap the value here before creating a
# timedelta object so the operation doesn't fail.
delay_backoff_in_seconds = min(
-delay.total_seconds() * (2 ** (self.try_number - 1)),
+modded_hash,
timedelta.max.total_seconds() - 1
)
delay = timedelta(seconds=delay_backoff_in_seconds)
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/258baf0f/tests/models.py
--
diff --git a/tests/models.py b/tests/models.py
index b3b8c2a..759be47 100644
--- a/tests/models.py
+++ b/tests/models.py
@@ -877,18 +877,25 @@ class TaskInstanceTest(unittest.TestCase):
owner='airflow',
start_date=datetime.datetime(2016, 2, 1, 0, 0, 0))
ti = TI(
-task=task, execution_date=datetime.datetime.now())
+task=task, execution_date=DEFAULT_DATE)
ti.end_date = datetime.datetime.now()
ti.try_number = 1
dt = ti.next_retry_datetime()
-self.assertEqual(dt, ti.end_date + delay)
+# between 30 * 2^0.5 and 30 * 2^1 (15 and 30)
+self.assertEqual(dt, ti.end_date + datetime.timedelta(seconds=20.0))
+
+ti.try_number = 4
+dt = ti.next_retry_datetime()
+# between 30 * 2^2 and 30 * 2^3 (120 and 240)
+self.assertEqual(dt, ti.end_date + datetime.timedelta(seconds=181.0))
ti.try_number = 6
dt = ti.next_retry_datetime()
-self.assertEqual(dt, ti.end_date + (2 ** 5) * delay)
+# between 30 * 2^4 and 30 * 2^5 (480 and 960)
+self.assertEqual(dt, ti.end_date + datetime.timedelta(seconds=825.0))
-ti.try_number = 8
+ti.try_number = 9
dt = ti.next_retry_datetime()
self.assertEqual(dt, ti.end_date+max_delay)
[04/18] incubator-airflow git commit: [AIRFLOW-1263] Dynamic height for charts
[AIRFLOW-1263] Dynamic height for charts
Dynamic heights for webserver charts so that
longer task
names fit
Closes #2344 from aoen/ddavydov--
dynamic_chart_heights
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/fc5fe5cc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/fc5fe5cc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/fc5fe5cc
Branch: refs/heads/v1-8-test
Commit: fc5fe5cc4e3b27058c777ff6036f5ea86f30adbd
Parents: 6600faf
Author: Dan Davydov
Authored: Mon Jun 5 16:32:42 2017 -0700
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/www/views.py | 25 -
1 file changed, 20 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/fc5fe5cc/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 29f3e91..46cbbb9 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -279,6 +279,16 @@ def should_hide_value_for_key(key_name):
and conf.getboolean('admin', 'hide_sensitive_variable_fields')
+
+def get_chart_height(dag):
+"""
+TODO(aoen): See [AIRFLOW-1263] We use the number of tasks in the DAG as a
heuristic to
+approximate the size of generated chart (otherwise the charts are tiny and
unreadable
+when DAGs have a large number of tasks). Ideally nvd3 should allow for
dynamic-height
+charts, that is charts that take up space based on the size of the
components within.
+"""
+return 600 + len(dag.tasks) * 10
+
class Airflow(BaseView):
def is_visible(self):
@@ -1389,10 +1399,12 @@ class Airflow(BaseView):
include_upstream=True,
include_downstream=False)
+
+chart_height = get_chart_height(dag)
chart = nvd3.lineChart(
-name="lineChart", x_is_date=True, height=600, width="1200")
+name="lineChart", x_is_date=True, height=chart_height,
width="1200")
cum_chart = nvd3.lineChart(
-name="cumLineChart", x_is_date=True, height=600, width="1200")
+name="cumLineChart", x_is_date=True, height=chart_height,
width="1200")
y = {}
x = {}
@@ -1483,8 +1495,10 @@ class Airflow(BaseView):
include_upstream=True,
include_downstream=False)
+chart_height = get_chart_height(dag)
chart = nvd3.lineChart(
-name="lineChart", x_is_date=True, y_axis_format='d', height=600,
width="1200")
+name="lineChart", x_is_date=True, y_axis_format='d',
height=chart_height,
+width="1200")
for task in dag.tasks:
y = []
@@ -1545,8 +1559,9 @@ class Airflow(BaseView):
include_upstream=True,
include_downstream=False)
+chart_height = get_chart_height(dag)
chart = nvd3.lineChart(
-name="lineChart", x_is_date=True, height=600, width="1200")
+name="lineChart", x_is_date=True, height=chart_height,
width="1200")
y = {}
x = {}
for task in dag.tasks:
@@ -1589,7 +1604,7 @@ class Airflow(BaseView):
'airflow/chart.html',
dag=dag,
chart=chart.htmlcontent,
-height="700px",
+height=str(chart_height + 100) + "px",
demo_mode=conf.getboolean('webserver', 'demo_mode'),
root=root,
form=form,
[13/18] incubator-airflow git commit: [AIRFLOW-1064] Change default sort to job_id for TaskInstanceModelView
[AIRFLOW-1064] Change default sort to job_id for TaskInstanceModelView
The TaskInstanceModelView default sort column is
on an unindexed column.
We shouldn't need an index on start_date, and
job_id is just as logical
of a default sort.
Closes #2215 from saguziel/aguziel-fix-ti-page
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/708e8ad7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/708e8ad7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/708e8ad7
Branch: refs/heads/v1-8-test
Commit: 708e8ad7bbaf865ca850bbcdfcfd89b914a09dea
Parents: c39d915
Author: Alex Guziel
Authored: Tue Apr 4 17:19:43 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/www/views.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/708e8ad7/airflow/www/views.py
--
diff --git a/airflow/www/views.py b/airflow/www/views.py
index 46cbbb9..a873e63 100644
--- a/airflow/www/views.py
+++ b/airflow/www/views.py
@@ -2355,7 +2355,7 @@ class TaskInstanceModelView(ModelViewOnly):
queued_dttm=datetime_f,
dag_id=dag_link, duration=duration_f)
column_searchable_list = ('dag_id', 'task_id', 'state')
-column_default_sort = ('start_date', True)
+column_default_sort = ('job_id', True)
form_choices = {
'state': [
('success', 'success'),
[17/18] incubator-airflow git commit: [AIRFLOW-XXX] Set version to 1.8.2rc1
[AIRFLOW-XXX] Set version to 1.8.2rc1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/7f7c52fe Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/7f7c52fe Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/7f7c52fe Branch: refs/heads/v1-8-test Commit: 7f7c52fe4f4ab93f9b073c69e83836caee4fb023 Parents: b0245e4 Author: Maxime Beauchemin Authored: Wed Jun 7 21:35:51 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 8 08:36:20 2017 -0700 -- CHANGELOG.txt | 72 - airflow/version.py | 2 +- 2 files changed, 36 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/7f7c52fe/CHANGELOG.txt -- diff --git a/CHANGELOG.txt b/CHANGELOG.txt index fe025c6..b5e9abf 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,43 +1,41 @@ +AIRFLOW 1.8.2rc1, 2017-06-05 +- + + AIRFLOW 1.8.1, 2017-05-09 - -[AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependencies Met -[AIRFLOW-1138] Add licenses to files in scripts directory -[AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE -[AIRFLOW-1124] Do not set all task instances to scheduled on backfill -[AIRFLOW-1120] Update version view to include Apache prefix -[AIRFLOW-1062] DagRun#find returns wrong result if external_trigger=False is specified -[AIRFLOW-1054] Fix broken import on test_dag -[AIRFLOW-1050] Retries ignored - regression -[AIRFLOW-1033] TypeError: can't compare datetime.datetime to None -[AIRFLOW-1017] get_task_instance should return None instead of throw an exception for non-existent TIs -[AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs -[AIRFLOW-1004] `airflow webserver -D` runs in foreground -[AIRFLOW-1001] Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'" on example_subdag_operator -[AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow -[AIRFLOW-989] Clear Task Regression -[AIRFLOW-974] airflow.util.file mkdir has a race condition -[AIRFLOW-906] Update Code icon from lightning bolt to file -[AIRFLOW-858] Configurable database name for DB operators -[AIRFLOW-853] ssh_execute_operator.py stdout decode default to ASCII -[AIRFLOW-832] Fix debug server -[AIRFLOW-817] Trigger dag fails when using CLI + API -[AIRFLOW-816] Make sure to pull nvd3 from local resources -[AIRFLOW-815] Add previous/next execution dates to available default variables. -[AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/job.py) -[AIRFLOW-812] Scheduler job terminates when there is no dag file -[AIRFLOW-806] UI should properly ignore DAG doc when it is None -[AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_CONN -[AIRFLOW-785] ImportError if cgroupspy is not installed -[AIRFLOW-784] Cannot install with funcsigs > 1.0.0 -[AIRFLOW-780] The UI no longer shows broken DAGs -[AIRFLOW-777] dag_is_running is initlialized to True instead of False -[AIRFLOW-719] Skipped operations make DAG finish prematurely -[AIRFLOW-694] Empty env vars do not overwrite non-empty config values -[AIRFLOW-492] Insert into dag_stats table results into failed task while task itself succeeded -[AIRFLOW-139] Executing VACUUM with PostgresOperator -[AIRFLOW-111] DAG concurrency is not honored -[AIRFLOW-88] Improve clarity Travis CI reports +[AIRFLOW-1160] Update Spark parameters for Mesos +[AIRFLOW 1149] Allow for custom filters in Jinja2 templates +[AIRFLOW-1119] Fix unload query so headers are on first row[] +[AIRFLOW-1089] Add Spark application arguments +[AIRFLOW-1078] Fix latest_runs endpoint for old flask versions +[AIRFLOW-1074] Don't count queued tasks for concurrency limits +[AIRFLOW-1064] Change default sort to job_id for TaskInstanceModelView +[AIRFLOW-1038] Specify celery serialization options explicitly +[AIRFLOW-1036] Randomize exponential backoff +[AIRFLOW-993] Update date inference logic +[AIRFLOW-1167] Support microseconds in FTPHook modification time +[AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery change +[AIRFLOW-1263] Dynamic height for charts +[AIRFLOW-1266] Increase width of gantt y axis +[AIRFLOW-1290] set docs author to 'Apache Airflow' +[AIRFLOW-1166] Speed up _change_state_for_tis_without_dagrun +[AIRFLOW-1192] Some enhancements to qubole_operator +[AIRFLOW-1281] Sort variables by key field by default +[AIRFLOW-1244] Forbid creation of a pool with empty name +[AIRFLOW-1243] DAGs table has no default entries to show +[AIRFLOW-1227] Remove empty column on the Logs view +[AIRFLOW-1226] Remove empty column on the Jobs view +[AIRFLOW-1199] Fix create modal +[AIRFLOW-1200] Forbid creation of a variable with an
[05/18] incubator-airflow git commit: [AIRFLOW-XXX] Updating CHANGELOG, README, and UPDATING after 1.8.1 release
[AIRFLOW-XXX] Updating CHANGELOG, README, and UPDATING after 1.8.1 release # Conflicts: # UPDATING.md Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/9627969e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/9627969e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/9627969e Branch: refs/heads/v1-8-test Commit: 9627969e68cba74c1622139fa08e6b905fbe64b8 Parents: 829a18a Author: Chris Riccomini Authored: Tue May 9 13:14:50 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 8 08:36:20 2017 -0700 -- CHANGELOG.txt | 41 + README.md | 4 +++- UPDATING.md | 22 ++ 3 files changed, 66 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9627969e/CHANGELOG.txt -- diff --git a/CHANGELOG.txt b/CHANGELOG.txt index 5048128..fe025c6 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,3 +1,44 @@ +AIRFLOW 1.8.1, 2017-05-09 +- + +[AIRFLOW-1142] SubDAG Tasks Not Executed Even Though All Dependencies Met +[AIRFLOW-1138] Add licenses to files in scripts directory +[AIRFLOW-1127] Move license notices to LICENSE instead of NOTICE +[AIRFLOW-1124] Do not set all task instances to scheduled on backfill +[AIRFLOW-1120] Update version view to include Apache prefix +[AIRFLOW-1062] DagRun#find returns wrong result if external_trigger=False is specified +[AIRFLOW-1054] Fix broken import on test_dag +[AIRFLOW-1050] Retries ignored - regression +[AIRFLOW-1033] TypeError: can't compare datetime.datetime to None +[AIRFLOW-1017] get_task_instance should return None instead of throw an exception for non-existent TIs +[AIRFLOW-1011] Fix bug in BackfillJob._execute() for SubDAGs +[AIRFLOW-1004] `airflow webserver -D` runs in foreground +[AIRFLOW-1001] Landing Time shows "unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'" on example_subdag_operator +[AIRFLOW-1000] Rebrand to Apache Airflow instead of Airflow +[AIRFLOW-989] Clear Task Regression +[AIRFLOW-974] airflow.util.file mkdir has a race condition +[AIRFLOW-906] Update Code icon from lightning bolt to file +[AIRFLOW-858] Configurable database name for DB operators +[AIRFLOW-853] ssh_execute_operator.py stdout decode default to ASCII +[AIRFLOW-832] Fix debug server +[AIRFLOW-817] Trigger dag fails when using CLI + API +[AIRFLOW-816] Make sure to pull nvd3 from local resources +[AIRFLOW-815] Add previous/next execution dates to available default variables. +[AIRFLOW-813] Fix unterminated unit tests in tests.job (tests/job.py) +[AIRFLOW-812] Scheduler job terminates when there is no dag file +[AIRFLOW-806] UI should properly ignore DAG doc when it is None +[AIRFLOW-794] Consistent access to DAGS_FOLDER and SQL_ALCHEMY_CONN +[AIRFLOW-785] ImportError if cgroupspy is not installed +[AIRFLOW-784] Cannot install with funcsigs > 1.0.0 +[AIRFLOW-780] The UI no longer shows broken DAGs +[AIRFLOW-777] dag_is_running is initlialized to True instead of False +[AIRFLOW-719] Skipped operations make DAG finish prematurely +[AIRFLOW-694] Empty env vars do not overwrite non-empty config values +[AIRFLOW-492] Insert into dag_stats table results into failed task while task itself succeeded +[AIRFLOW-139] Executing VACUUM with PostgresOperator +[AIRFLOW-111] DAG concurrency is not honored +[AIRFLOW-88] Improve clarity Travis CI reports + AIRFLOW 1.8.0, 2017-03-12 - http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/9627969e/README.md -- diff --git a/README.md b/README.md index fc768a7..57f9aea 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Airflow -[](https://badge.fury.io/py/airflow) +[](https://badge.fury.io/py/apache-airflow) [](https://travis-ci.org/apache/incubator-airflow) [](https://codecov.io/github/apache/incubator-airflow?branch=master) [](https://landscape.io/github/apache/incubator-airflow/master) @@ -8,6 +8,8 @@ [](http://pythonhosted.org/airflow/) [](https://gitter.im/apache/incubator-airflow?utm_source=b
[09/18] incubator-airflow git commit: [AIRFLOW-1167] Support microseconds in FTPHook modification time
[AIRFLOW-1167] Support microseconds in FTPHook modification time
Closes #2268 from NielsZeilemaker/fix-ftp-hook
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c5d6c3a3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c5d6c3a3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c5d6c3a3
Branch: refs/heads/v1-8-test
Commit: c5d6c3a3cbd5ad28470e2d7e8434f439aae14a75
Parents: 1b2b34e
Author: Niels Zeilemaker
Authored: Fri May 5 09:17:40 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/contrib/hooks/ftp_hook.py| 7 ++-
tests/contrib/hooks/test_ftp_hook.py | 18 ++
2 files changed, 24 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c5d6c3a3/airflow/contrib/hooks/ftp_hook.py
--
diff --git a/airflow/contrib/hooks/ftp_hook.py
b/airflow/contrib/hooks/ftp_hook.py
index 2f2e4c2..dbd353b 100644
--- a/airflow/contrib/hooks/ftp_hook.py
+++ b/airflow/contrib/hooks/ftp_hook.py
@@ -226,7 +226,12 @@ class FTPHook(BaseHook):
def get_mod_time(self, path):
conn = self.get_conn()
ftp_mdtm = conn.sendcmd('MDTM ' + path)
-return datetime.datetime.strptime(ftp_mdtm[4:], '%Y%m%d%H%M%S')
+time_val = ftp_mdtm[4:]
+# time_val optionally has microseconds
+try:
+return datetime.datetime.strptime(time_val, "%Y%m%d%H%M%S.%f")
+except ValueError:
+return datetime.datetime.strptime(time_val, '%Y%m%d%H%M%S')
class FTPSHook(FTPHook):
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c5d6c3a3/tests/contrib/hooks/test_ftp_hook.py
--
diff --git a/tests/contrib/hooks/test_ftp_hook.py
b/tests/contrib/hooks/test_ftp_hook.py
index ab6f459..c81593d 100644
--- a/tests/contrib/hooks/test_ftp_hook.py
+++ b/tests/contrib/hooks/test_ftp_hook.py
@@ -77,6 +77,24 @@ class TestFTPHook(unittest.TestCase):
self.conn_mock.rename.assert_called_once_with(from_path, to_path)
self.conn_mock.quit.assert_called_once_with()
+
+def test_mod_time(self):
+self.conn_mock.sendcmd.return_value = '213 20170428010138'
+
+path = '/path/file'
+with fh.FTPHook() as ftp_hook:
+ftp_hook.get_mod_time(path)
+
+self.conn_mock.sendcmd.assert_called_once_with('MDTM ' + path)
+
+def test_mod_time_micro(self):
+self.conn_mock.sendcmd.return_value = '213 20170428010138.003'
+
+path = '/path/file'
+with fh.FTPHook() as ftp_hook:
+ftp_hook.get_mod_time(path)
+
+self.conn_mock.sendcmd.assert_called_once_with('MDTM ' + path)
if __name__ == '__main__':
[10/18] incubator-airflow git commit: [AIRFLOW-993] Update date inference logic
[AIRFLOW-993] Update date inference logic
DAGs should set task start_date and end_date when
possible, making sure
they agree with the DAGâs own dates.
Closes #2157 from jlowin/run-bug
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/65039959
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/65039959
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/65039959
Branch: refs/heads/v1-8-test
Commit: 650399590b06b11b67bbda699947574e50faed38
Parents: c5d6c3a
Author: Jeremiah Lowin
Authored: Sat May 13 14:53:08 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/models.py | 15 ++-
tests/models.py | 39 +++
2 files changed, 53 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/65039959/airflow/models.py
--
diff --git a/airflow/models.py b/airflow/models.py
index 484cb8c..bf50c4c 100755
--- a/airflow/models.py
+++ b/airflow/models.py
@@ -3269,8 +3269,21 @@ class DAG(BaseDag, LoggingMixin):
"""
if not self.start_date and not task.start_date:
raise AirflowException("Task is missing the start_date parameter")
-if not task.start_date:
+# if the task has no start date, assign it the same as the DAG
+elif not task.start_date:
task.start_date = self.start_date
+# otherwise, the task will start on the later of its own start date and
+# the DAG's start date
+elif self.start_date:
+task.start_date = max(task.start_date, self.start_date)
+
+# if the task has no end date, assign it the same as the dag
+if not task.end_date:
+task.end_date = self.end_date
+# otherwise, the task will end on the earlier of its own end date and
+# the DAG's end date
+elif task.end_date and self.end_date:
+task.end_date = min(task.end_date, self.end_date)
if task.task_id in self.task_dict:
# TODO: raise an error in Airflow 2.0
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/65039959/tests/models.py
--
diff --git a/tests/models.py b/tests/models.py
index 392de03..b3b8c2a 100644
--- a/tests/models.py
+++ b/tests/models.py
@@ -575,6 +575,45 @@ class DagBagTest(unittest.TestCase):
class TaskInstanceTest(unittest.TestCase):
+def test_set_task_dates(self):
+"""
+Test that tasks properly take start/end dates from DAGs
+"""
+dag = DAG('dag', start_date=DEFAULT_DATE, end_date=DEFAULT_DATE +
datetime.timedelta(days=10))
+
+op1 = DummyOperator(task_id='op_1', owner='test')
+
+self.assertTrue(op1.start_date is None and op1.end_date is None)
+
+# dag should assign its dates to op1 because op1 has no dates
+dag.add_task(op1)
+self.assertTrue(
+op1.start_date == dag.start_date and op1.end_date == dag.end_date)
+
+op2 = DummyOperator(
+task_id='op_2',
+owner='test',
+start_date=DEFAULT_DATE - datetime.timedelta(days=1),
+end_date=DEFAULT_DATE + datetime.timedelta(days=11))
+
+# dag should assign its dates to op2 because they are more restrictive
+dag.add_task(op2)
+self.assertTrue(
+op2.start_date == dag.start_date and op2.end_date == dag.end_date)
+
+op3 = DummyOperator(
+task_id='op_3',
+owner='test',
+start_date=DEFAULT_DATE + datetime.timedelta(days=1),
+end_date=DEFAULT_DATE + datetime.timedelta(days=9))
+# op3 should keep its dates because they are more restrictive
+dag.add_task(op3)
+self.assertTrue(
+op3.start_date == DEFAULT_DATE + datetime.timedelta(days=1))
+self.assertTrue(
+op3.end_date == DEFAULT_DATE + datetime.timedelta(days=9))
+
+
def test_set_dag(self):
"""
Test assigning Operators to Dags, including deferred assignment
[07/18] incubator-airflow git commit: [AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery change
[AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery change Closes #2279 from NielsZeilemaker/AIRFLOW-1179 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/1b2b34e6 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/1b2b34e6 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/1b2b34e6 Branch: refs/heads/v1-8-test Commit: 1b2b34e6a0b44b5a686cae6fa6e4014705769d67 Parents: fc5fe5c Author: Niels Zeilemaker Authored: Tue May 9 09:42:32 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 8 08:36:20 2017 -0700 -- airflow/contrib/hooks/bigquery_hook.py | 2 +- scripts/ci/requirements.txt| 1 + setup.py | 1 + 3 files changed, 3 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1b2b34e6/airflow/contrib/hooks/bigquery_hook.py -- diff --git a/airflow/contrib/hooks/bigquery_hook.py b/airflow/contrib/hooks/bigquery_hook.py index 53ca123..06de4e8 100644 --- a/airflow/contrib/hooks/bigquery_hook.py +++ b/airflow/contrib/hooks/bigquery_hook.py @@ -24,7 +24,7 @@ import time from apiclient.discovery import build, HttpError from googleapiclient import errors from builtins import range -from pandas.io.gbq import GbqConnector, \ +from pandas_gbq.gbq import GbqConnector, \ _parse_data as gbq_parse_data, \ _check_google_client_version as gbq_check_google_client_version, \ _test_google_api_imports as gbq_test_google_api_imports http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1b2b34e6/scripts/ci/requirements.txt -- diff --git a/scripts/ci/requirements.txt b/scripts/ci/requirements.txt index 6c65167..813d035 100644 --- a/scripts/ci/requirements.txt +++ b/scripts/ci/requirements.txt @@ -53,6 +53,7 @@ nose-exclude nose-ignore-docstring==0.2 nose-parameterized pandas +pandas-gbq psutil>=4.2.0, <5.0.0 psycopg2 pydruid http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/1b2b34e6/setup.py -- diff --git a/setup.py b/setup.py index 288c2b3..2b1ac2f 100644 --- a/setup.py +++ b/setup.py @@ -137,6 +137,7 @@ gcp_api = [ 'google-api-python-client>=1.5.0, <1.6.0', 'oauth2client>=2.0.2, <2.1.0', 'PyOpenSSL', +'pandas-gbq' ] hdfs = ['snakebite>=2.7.8'] webhdfs = ['hdfs[dataframe,avro,kerberos]>=2.0.4']
[12/18] incubator-airflow git commit: [AIRFLOW-1038] Specify celery serialization options explicitly
[AIRFLOW-1038] Specify celery serialization options explicitly
Specify the CELERY_TASK_SERIALIZER and CELERY_RESULT_SERIALIZER as
pickle explicitly, and CELERY_EVENT_SERIALIZER as json.
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/c39d9153
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/c39d9153
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/c39d9153
Branch: refs/heads/v1-8-test
Commit: c39d91537bb8c28cc81aad3e26eff72158e3b1f4
Parents: 258baf0
Author: Alex Guziel
Authored: Fri Mar 24 11:51:39 2017 -0700
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/executors/celery_executor.py | 3 +++
1 file changed, 3 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/c39d9153/airflow/executors/celery_executor.py
--
diff --git a/airflow/executors/celery_executor.py
b/airflow/executors/celery_executor.py
index 04414fb..e0c94c1 100644
--- a/airflow/executors/celery_executor.py
+++ b/airflow/executors/celery_executor.py
@@ -36,6 +36,9 @@ DEFAULT_QUEUE = configuration.get('celery', 'DEFAULT_QUEUE')
class CeleryConfig(object):
CELERY_ACCEPT_CONTENT = ['json', 'pickle']
+CELERY_EVENT_SERIALIZER = 'json'
+CELERY_RESULT_SERIALIZER = 'pickle'
+CELERY_TASK_SERIALIZER = 'pickle'
CELERYD_PREFETCH_MULTIPLIER = 1
CELERY_ACKS_LATE = True
BROKER_URL = configuration.get('celery', 'BROKER_URL')
[08/18] incubator-airflow git commit: [AIRFLOW-1266] Increase width of gantt y axis
[AIRFLOW-1266] Increase width of gantt y axis
Increase the width of the gantt view y axis to
accommodate larger
task names.
Closes #2345 from aoen/ddavydov--
increase_width_of_gantt_y_axis
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/6600faf0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/6600faf0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/6600faf0
Branch: refs/heads/v1-8-test
Commit: 6600faf03a99eeb68138c88539bf847bdea2f32c
Parents: f8f4e60
Author: Dan Davydov
Authored: Mon Jun 5 16:31:42 2017 -0700
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/www/static/gantt-chart-d3v2.js | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/6600faf0/airflow/www/static/gantt-chart-d3v2.js
--
diff --git a/airflow/www/static/gantt-chart-d3v2.js
b/airflow/www/static/gantt-chart-d3v2.js
index 2aee016..d21311a 100644
--- a/airflow/www/static/gantt-chart-d3v2.js
+++ b/airflow/www/static/gantt-chart-d3v2.js
@@ -48,6 +48,7 @@ d3.gantt = function() {
bottom : 20,
left : 150
};
+ var yAxisLeftOffset = 220;
var selector = 'body';
var timeDomainStart = d3.time.day.offset(new Date(),-3);
var timeDomainEnd = d3.time.hour.offset(new Date(),+3);
@@ -64,10 +65,10 @@ d3.gantt = function() {
};
var rectTransform = function(d) {
-return "translate(" + x(d.startDate) + "," + y(d.taskName) + ")";
+return "translate(" + (x(d.startDate) + yAxisLeftOffset) + "," +
y(d.taskName) + ")";
};
- var x = d3.time.scale().domain([ timeDomainStart, timeDomainEnd ]).range([
0, width ]).clamp(true);
+ var x = d3.time.scale().domain([ timeDomainStart, timeDomainEnd ]).range([
0, (width-yAxisLeftOffset) ]).clamp(true);
var y = d3.scale.ordinal().domain(taskTypes).rangeRoundBands([ 0, height -
margin.top - margin.bottom ], .1);
@@ -95,7 +96,7 @@ d3.gantt = function() {
};
var initAxis = function() {
-x = d3.time.scale().domain([ timeDomainStart, timeDomainEnd ]).range([ 0,
width ]).clamp(true);
+x = d3.time.scale().domain([ timeDomainStart, timeDomainEnd ]).range([ 0,
width-yAxisLeftOffset ]).clamp(true);
y = d3.scale.ordinal().domain(taskTypes).rangeRoundBands([ 0, height -
margin.top - margin.bottom ], .1);
xAxis =
d3.svg.axis().scale(x).orient("bottom").tickFormat(d3.time.format(tickFormat)).tickSubdivide(true)
.tickSize(8).tickPadding(8);
@@ -141,11 +142,11 @@ d3.gantt = function() {
svg.append("g")
.attr("class", "x axis")
-.attr("transform", "translate(0, " + (height - margin.top - margin.bottom)
+ ")")
+.attr("transform", "translate(" + yAxisLeftOffset + ", " + (height -
margin.top - margin.bottom) + ")")
.transition()
.call(xAxis);
-svg.append("g").attr("class", "y axis").transition().call(yAxis);
+svg.append("g").attr("class", "y axis").transition().attr("transform",
"translate(" + yAxisLeftOffset + ", 0)").call(yAxis);
svg.call(tip);
return gantt;
[03/18] incubator-airflow git commit: [AIRFLOW-1078] Fix latest_runs endpoint for old flask versions
[AIRFLOW-1078] Fix latest_runs endpoint for old flask versions
Old versions of flask (<0.11) dont support jsonify
on arrays due an
ECMAScript 4 vulnerability in older browsers. This
should work on old
flask versions as well.
Closes #2224 from saguziel/aguziel-fix-homepage
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/156e90b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/156e90b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/156e90b5
Branch: refs/heads/v1-8-test
Commit: 156e90b5c176633588fa7e5021de0185b583b8fc
Parents: 7002493
Author: Alex Guziel
Authored: Fri Apr 7 19:26:35 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/www/api/experimental/endpoints.py | 2 +-
airflow/www/templates/airflow/dags.html | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/156e90b5/airflow/www/api/experimental/endpoints.py
--
diff --git a/airflow/www/api/experimental/endpoints.py
b/airflow/www/api/experimental/endpoints.py
index 63355c7..2dacd6c 100644
--- a/airflow/www/api/experimental/endpoints.py
+++ b/airflow/www/api/experimental/endpoints.py
@@ -130,4 +130,4 @@ def latest_dag_runs():
'dag_run_url': url_for('airflow.graph', dag_id=dagrun.dag_id,
execution_date=dagrun.execution_date)
})
-return jsonify(payload)
+return jsonify(items=payload) # old flask versions dont support jsonifying
arrays
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/156e90b5/airflow/www/templates/airflow/dags.html
--
diff --git a/airflow/www/templates/airflow/dags.html
b/airflow/www/templates/airflow/dags.html
index e0546d6..16bbfb0 100644
--- a/airflow/www/templates/airflow/dags.html
+++ b/airflow/www/templates/airflow/dags.html
@@ -227,7 +227,7 @@
});
});
$.getJSON("{{ url_for('api_experimental.latest_dag_runs') }}",
function(data) {
-$.each(data, function() {
+$.each(data["items"], function() {
var link = $("", {
href: this.dag_run_url,
text: this.execution_date
[15/18] incubator-airflow git commit: [AIRFLOW-1074] Don't count queued tasks for concurrency limits
[AIRFLOW-1074] Don't count queued tasks for concurrency limits There may be orphaned tasks queued but not in a running dag run that will not cleared. We should not count these as they will interfere. I hate to do this, but I changed my mind on counting queued tasks. 1. Queued tasks that are actually queued generally get set to running pretty quickly. 2. Because of the worker-side check, we won't actually pass concurrency. I don't think the queued thing is a big deal because of this, I'm more worried about orphaned tasks that are in QUEUED state but not in a running dag_run (so they wont get reset) interfering with concurrency. There may be orphaned tasks queued but not in a running dag run that will not cleared. We should not count these as they will interfere. Closes #2221 from saguziel/aguziel-concurrency-2 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/70024935 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/70024935 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/70024935 Branch: refs/heads/v1-8-test Commit: 70024935f24e0ff3d2861c0ccfa69cdd38084b9d Parents: 708e8ad Author: Alex Guziel Authored: Wed Apr 12 11:56:03 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 8 08:36:20 2017 -0700 -- airflow/jobs.py | 3 ++- tests/jobs.py | 6 +++--- 2 files changed, 5 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/70024935/airflow/jobs.py -- diff --git a/airflow/jobs.py b/airflow/jobs.py index edc7458..457966f 100644 --- a/airflow/jobs.py +++ b/airflow/jobs.py @@ -1062,11 +1062,12 @@ class SchedulerJob(BaseJob): dag_id = task_instance.dag_id if dag_id not in dag_id_to_possibly_running_task_count: +# TODO(saguziel): also check against QUEUED state, see AIRFLOW-1104 dag_id_to_possibly_running_task_count[dag_id] = \ DAG.get_num_task_instances( dag_id, simple_dag_bag.get_dag(dag_id).task_ids, -states=[State.RUNNING, State.QUEUED], +states=[State.RUNNING], session=session) current_task_concurrency = dag_id_to_possibly_running_task_count[dag_id] http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/70024935/tests/jobs.py -- diff --git a/tests/jobs.py b/tests/jobs.py index 428d3ac..21102e6 100644 --- a/tests/jobs.py +++ b/tests/jobs.py @@ -573,14 +573,14 @@ class SchedulerJobTest(unittest.TestCase): ti1.refresh_from_db() ti2.refresh_from_db() ti1.state = State.RUNNING -ti2.state = State.QUEUED +ti2.state = State.RUNNING session.merge(ti1) session.merge(ti2) session.commit() self.assertEqual(State.RUNNING, dr1.state) self.assertEqual(2, DAG.get_num_task_instances(dag_id, dag.task_ids, -states=[State.RUNNING, State.QUEUED], session=session)) +states=[State.RUNNING], session=session)) # create second dag run dr2 = scheduler.create_dag_run(dag) @@ -607,7 +607,7 @@ class SchedulerJobTest(unittest.TestCase): self.assertEqual(3, DAG.get_num_task_instances(dag_id, dag.task_ids, states=[State.RUNNING, State.QUEUED], session=session)) self.assertEqual(State.RUNNING, ti1.state) -self.assertEqual(State.QUEUED, ti2.state) +self.assertEqual(State.RUNNING, ti2.state) six.assertCountEqual(self, [State.QUEUED, State.SCHEDULED], [ti3.state, ti4.state]) session.close()
[06/18] incubator-airflow git commit: [AIRFLOW-1290] set docs author to 'Apache Airflow'
[AIRFLOW-1290] set docs author to 'Apache Airflow'
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f8f4e605
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f8f4e605
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f8f4e605
Branch: refs/heads/v1-8-test
Commit: f8f4e605c9adc1edac414178d0cd5b3fc8f49adc
Parents: 9627969
Author: Maxime Beauchemin
Authored: Wed Jun 7 13:54:00 2017 -0700
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
docs/conf.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f8f4e605/docs/conf.py
--
diff --git a/docs/conf.py b/docs/conf.py
index c462847..6c84fcd 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -224,7 +224,7 @@ latex_elements = {
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'Airflow.tex', u'Airflow Documentation',
- u'Maxime Beauchemin', 'manual'),
+ u'Apache Airflow', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
@@ -254,7 +254,7 @@ latex_documents = [
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'airflow', u'Airflow Documentation',
- [u'Maxime Beauchemin'], 1)
+ [u'Apache Airflow'], 1)
]
# If true, show URL addresses after external links.
@@ -268,7 +268,7 @@ man_pages = [
# dir menu entry, description, category)
texinfo_documents = [(
'index', 'Airflow', u'Airflow Documentation',
-u'Maxime Beauchemin', 'Airflow',
+u'Apache Airflow', 'Airflow',
'Airflow is a system to programmaticaly author, schedule and monitor data
pipelines.',
'Miscellaneous'
),]
[02/18] incubator-airflow git commit: [AIRFLOW-1160] Update Spark parameters for Mesos
[AIRFLOW-1160] Update Spark parameters for Mesos
Closes #2265 from cameres/master
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b0245e4a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b0245e4a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b0245e4a
Branch: refs/heads/v1-8-test
Commit: b0245e4a3dacf665e26446acafb3e80f558a83e0
Parents: 9744b1a
Author: Connor Ameres
Authored: Mon May 1 23:22:04 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/contrib/hooks/spark_sql_hook.py| 8 +++-
airflow/contrib/hooks/spark_submit_hook.py | 8 +++-
airflow/contrib/operators/spark_sql_operator.py| 7 ++-
airflow/contrib/operators/spark_submit_operator.py | 7 ++-
tests/contrib/hooks/spark_submit_hook.py | 2 ++
tests/contrib/operators/spark_submit_operator.py | 2 ++
6 files changed, 30 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b0245e4a/airflow/contrib/hooks/spark_sql_hook.py
--
diff --git a/airflow/contrib/hooks/spark_sql_hook.py
b/airflow/contrib/hooks/spark_sql_hook.py
index 37df3fe..8d73f60 100644
--- a/airflow/contrib/hooks/spark_sql_hook.py
+++ b/airflow/contrib/hooks/spark_sql_hook.py
@@ -31,7 +31,9 @@ class SparkSqlHook(BaseHook):
:type conf: str (format: PROP=VALUE)
:param conn_id: connection_id string
:type conn_id: str
-:param executor_cores: Number of cores per executor
+:param total_executor_cores: (Standalone & Mesos only) Total cores for all
executors (Default: all the available cores on the worker)
+:type total_executor_cores: int
+:param executor_cores: (Standalone & YARN only) Number of cores per
executor (Default: 2)
:type executor_cores: int
:param executor_memory: Memory per executor (e.g. 1000M, 2G) (Default: 1G)
:type executor_memory: str
@@ -52,6 +54,7 @@ class SparkSqlHook(BaseHook):
sql,
conf=None,
conn_id='spark_sql_default',
+ total_executor_cores=None,
executor_cores=None,
executor_memory=None,
keytab=None,
@@ -64,6 +67,7 @@ class SparkSqlHook(BaseHook):
self._sql = sql
self._conf = conf
self._conn = self.get_connection(conn_id)
+self._total_executor_cores = total_executor_cores
self._executor_cores = executor_cores
self._executor_memory = executor_memory
self._keytab = keytab
@@ -89,6 +93,8 @@ class SparkSqlHook(BaseHook):
if self._conf:
for conf_el in self._conf.split(","):
connection_cmd += ["--conf", conf_el]
+if self._total_executor_cores:
+connection_cmd += ["--total-executor-cores",
str(self._total_executor_cores)]
if self._executor_cores:
connection_cmd += ["--executor-cores", str(self._executor_cores)]
if self._executor_memory:
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/b0245e4a/airflow/contrib/hooks/spark_submit_hook.py
--
diff --git a/airflow/contrib/hooks/spark_submit_hook.py
b/airflow/contrib/hooks/spark_submit_hook.py
index e4ce797..c34538e 100644
--- a/airflow/contrib/hooks/spark_submit_hook.py
+++ b/airflow/contrib/hooks/spark_submit_hook.py
@@ -42,7 +42,9 @@ class SparkSubmitHook(BaseHook):
:type jars: str
:param java_class: the main class of the Java application
:type java_class: str
-:param executor_cores: Number of cores per executor (Default: 2)
+:param total_executor_cores: (Standalone & Mesos only) Total cores for all
executors (Default: all the available cores on the worker)
+:type total_executor_cores: int
+:param executor_cores: (Standalone & YARN only) Number of cores per
executor (Default: 2)
:type executor_cores: int
:param executor_memory: Memory per executor (e.g. 1000M, 2G) (Default: 1G)
:type executor_memory: str
@@ -69,6 +71,7 @@ class SparkSubmitHook(BaseHook):
py_files=None,
jars=None,
java_class=None,
+ total_executor_cores=None,
executor_cores=None,
executor_memory=None,
driver_memory=None,
@@ -84,6 +87,7 @@ class SparkSubmitHook(BaseHook):
self._py_files = py_files
self._jars = jars
self._java_class = java_class
+self._total_executor_cores = total_executor_cores
self._executor_cores = executor_cores
self._executor_memory = executor_memory
[16/18] incubator-airflow git commit: [AIRFLOW-1119] Fix unload query so headers are on first row[]
[AIRFLOW-1119] Fix unload query so headers are on first row[]
Closes #2245 from th11/airflow-1119-fix
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/07ebc46e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/07ebc46e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/07ebc46e
Branch: refs/heads/v1-8-test
Commit: 07ebc46ea7237f554477f83575ec65df95b321f0
Parents: 34f072a
Author: Thomas Hofer
Authored: Tue Apr 25 11:31:31 2017 +0200
Committer: Maxime Beauchemin
Committed: Thu Jun 8 08:36:20 2017 -0700
--
airflow/operators/redshift_to_s3_operator.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/07ebc46e/airflow/operators/redshift_to_s3_operator.py
--
diff --git a/airflow/operators/redshift_to_s3_operator.py
b/airflow/operators/redshift_to_s3_operator.py
index d9ef59d..fda88d9 100644
--- a/airflow/operators/redshift_to_s3_operator.py
+++ b/airflow/operators/redshift_to_s3_operator.py
@@ -93,7 +93,8 @@ class RedshiftToS3Transfer(BaseOperator):
unload_query = """
UNLOAD ('SELECT {0}
UNION ALL
-SELECT {1} FROM {2}.{3}')
+SELECT {1} FROM {2}.{3}
+ORDER BY 1 DESC')
TO 's3://{4}/{5}/{3}_'
with
credentials
'aws_access_key_id={6};aws_secret_access_key={7}'
[01/18] incubator-airflow git commit: Update CHANGELOG for 1.8.2rc1
Repository: incubator-airflow Updated Branches: refs/heads/v1-8-test 829a18a4b -> a2dd2465a Update CHANGELOG for 1.8.2rc1 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/a2dd2465 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/a2dd2465 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/a2dd2465 Branch: refs/heads/v1-8-test Commit: a2dd2465ae37cd6bba20d5f127a29be0c5080fc4 Parents: 7f7c52f Author: Maxime Beauchemin Authored: Wed Jun 7 21:37:28 2017 -0700 Committer: Maxime Beauchemin Committed: Thu Jun 8 08:36:20 2017 -0700 -- CHANGELOG.txt | 33 + 1 file changed, 33 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/a2dd2465/CHANGELOG.txt -- diff --git a/CHANGELOG.txt b/CHANGELOG.txt index b5e9abf..8d59fa1 100644 --- a/CHANGELOG.txt +++ b/CHANGELOG.txt @@ -1,6 +1,39 @@ AIRFLOW 1.8.2rc1, 2017-06-05 - +e10af9a [AIRFLOW-XXX] Set version to 1.8.2rc1 +69bd269 [AIRFLOW-1160] Update Spark parameters for Mesos +9692510 [AIRFLOW 1149][AIRFLOW-1149] Allow for custom filters in Jinja2 templates +6de5330 [AIRFLOW-1119] Fix unload query so headers are on first row[] +b4e9eb8 [AIRFLOW-1089] Add Spark application arguments +a4083f3 [AIRFLOW-1078] Fix latest_runs endpoint for old flask versions +7a02841 [AIRFLOW-1074] Don't count queued tasks for concurrency limits +a2c18a5 [AIRFLOW-1064] Change default sort to job_id for TaskInstanceModelView +d1c64ab [AIRFLOW-1038] Specify celery serialization options explicitly +b4ee88a [AIRFLOW-1036] Randomize exponential backoff +9fca409 [AIRFLOW-993] Update date inference logic +272c2f5 [AIRFLOW-1167] Support microseconds in FTPHook modification time +c7c0b72 [AIRFLOW-1179] Fix Pandas 0.2x breaking Google BigQuery change +acd0166 [AIRFLOW-1263] Dynamic height for charts +7f33f6e [AIRFLOW-1266] Increase width of gantt y axis +fc33c04 [AIRFLOW-1290] set docs author to 'Apache Airflow' +2e9eee3 [AIRFLOW-1282] Fix known event column sorting +2389a8a [AIRFLOW-1166] Speed up _change_state_for_tis_without_dagrun +bf966e6 [AIRFLOW-1192] Some enhancements to qubole_operator +57d5bcd [AIRFLOW-1281] Sort variables by key field by default +802fc15 [AIRFLOW-1244] Forbid creation of a pool with empty name +1232b6a [AIRFLOW-1243] DAGs table has no default entries to show +b0ba3c9 [AIRFLOW-1227] Remove empty column on the Logs view +c406652 [AIRFLOW-1226] Remove empty column on the Jobs view +51a83cc [AIRFLOW-1199] Fix create modal +cac7d4c [AIRFLOW-1200] Forbid creation of a variable with an empty key +5f3ee52 [AIRFLOW-1186] Sort dag.get_task_instances by execution_date +f446c08 [AIRFLOW-1145] Fix closest_date_partition function with before set to True If we're looking for the closest date before, we should take the latest date in the list of date before. +93b8e96 [AIRFLOW-1180] Fix flask-wtf version for test_csrf_rejection +bb56805 [AIRFLOW-1170] DbApiHook insert_rows inserts parameters separately +093b2f0 [AIRFLOW-1150] Fix scripts execution in sparksql hook[] +777f181 [AIRFLOW-1168] Add closing() to all connections and cursors +bc8e912 [AIRFLOW-XXX] Updating CHANGELOG, README, and UPDATING after 1.8.1 release AIRFLOW 1.8.1, 2017-05-09 -
[2/2] incubator-airflow git commit: Merge pull request #2352 from mistercrunch/remove_max_author
Merge pull request #2352 from mistercrunch/remove_max_author Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/b13cd6db Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/b13cd6db Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/b13cd6db Branch: refs/heads/master Commit: b13cd6db989e13bb23424d03a6acff79ee279693 Parents: ffe6162 92792e4 Author: Maxime Beauchemin Authored: Wed Jun 7 14:26:36 2017 -0700 Committer: Maxime Beauchemin Committed: Wed Jun 7 14:26:36 2017 -0700 -- docs/conf.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) --
[1/2] incubator-airflow git commit: [AIRFLOW-1290] set docs author to 'Apache Airflow'
Repository: incubator-airflow
Updated Branches:
refs/heads/master ffe6162f2 -> b13cd6db9
[AIRFLOW-1290] set docs author to 'Apache Airflow'
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/92792e47
Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/92792e47
Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/92792e47
Branch: refs/heads/master
Commit: 92792e472cf5c5084bb43b9549772a37e3c61149
Parents: ffe6162
Author: Maxime Beauchemin
Authored: Wed Jun 7 13:54:00 2017 -0700
Committer: Maxime Beauchemin
Committed: Wed Jun 7 13:54:00 2017 -0700
--
docs/conf.py | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/92792e47/docs/conf.py
--
diff --git a/docs/conf.py b/docs/conf.py
index c462847..6c84fcd 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -224,7 +224,7 @@ latex_elements = {
# author, documentclass [howto, manual, or own class]).
latex_documents = [
('index', 'Airflow.tex', u'Airflow Documentation',
- u'Maxime Beauchemin', 'manual'),
+ u'Apache Airflow', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
@@ -254,7 +254,7 @@ latex_documents = [
# (source start file, name, description, authors, manual section).
man_pages = [
('index', 'airflow', u'Airflow Documentation',
- [u'Maxime Beauchemin'], 1)
+ [u'Apache Airflow'], 1)
]
# If true, show URL addresses after external links.
@@ -268,7 +268,7 @@ man_pages = [
# dir menu entry, description, category)
texinfo_documents = [(
'index', 'Airflow', u'Airflow Documentation',
-u'Maxime Beauchemin', 'Airflow',
+u'Apache Airflow', 'Airflow',
'Airflow is a system to programmaticaly author, schedule and monitor data
pipelines.',
'Miscellaneous'
),]
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/readthedocs-test [created] 9dc8455fa
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.8.0-docs a86c7674d -> 37262bc7d
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.8.0-docs [created] a86c7674d
[1/2] incubator-airflow-site git commit: v1.8.0
Repository: incubator-airflow-site
Updated Branches:
refs/heads/asf-site 5e5740122 -> 69cff4922
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/69cff492/searchindex.js
--
diff --git a/searchindex.js b/searchindex.js
index 625cb26..121f5a8 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({docnames:["api","cli","code","concepts","configuration","faq","index","installation","integration","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],envversion:50,filenames:["api.rst","cli.rst","code.rst","concepts.rst","configuration.rst","faq.rst","index.rst","installation.rst","integration.rst","license.rst","plugins.rst","profiling.rst","project.rst","scheduler.rst","security.rst","start.rst","tutorial.rst","ui.rst"],objects:{"airflow.contrib":{hooks:[2,1,0,"-"],operators:[2,1,0,"-"]},"airflow.contrib.executors.mesos_executor":{MesosExecutor:[2,0,1,""]},"airflow.contrib.hooks":{FTPHook:[2,0,1,""],SSHHook:[2,0,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[2,2,1,""],create_directory:[2,2,1,""],delete_directory:[2,2,1,""],delete_file:[2,2,1,""],describe_directory:[2,2,1,""],get_conn:[2,2,1,""],list_directory:[2,2,1,""],rename:[2,2,1,""],retrieve_file:[2,2,1,""],store_file:[2,2,1,""]},"airflow.contrib.hooks.SSHHook":{Pop
en:[2,2,1,""],check_output:[2,2,1,""],tunnel:[2,2,1,""]},"airflow.contrib.operators":{SSHExecuteOperator:[2,0,1,""]},"airflow.contrib.operators.hipchat_operator":{HipChatAPIOperator:[2,0,1,""],HipChatAPISendRoomNotificationOperator:[2,0,1,""]},"airflow.executors":{LocalExecutor:[2,0,1,""],SequentialExecutor:[2,0,1,""]},"airflow.hooks":{DbApiHook:[2,0,1,""],HttpHook:[2,0,1,""],MySqlHook:[2,0,1,""],PrestoHook:[2,0,1,""],SqliteHook:[2,0,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[2,2,1,""],bulk_load:[2,2,1,""],get_conn:[2,2,1,""],get_cursor:[2,2,1,""],get_first:[2,2,1,""],get_pandas_df:[2,2,1,""],get_records:[2,2,1,""],insert_rows:[2,2,1,""],run:[2,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[2,2,1,""],run:[2,2,1,""],run_and_check:[2,2,1,""]},"airflow.hooks.MySqlHook":{bulk_load:[2,2,1,""],get_conn:[2,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[2,2,1,""],get_first:[2,2,1,""],get_pandas_df:[2,2,1,""],get_records:[2,2,1,""],run:[2,2,1,""]},"airflow.hooks.SqliteHook":{get_conn:[2,2,1,
""]},"airflow.macros":{ds_add:[2,3,1,""],ds_format:[2,3,1,""],hive:[2,1,0,"-"],random:[2,3,1,""]},"airflow.macros.hive":{closest_ds_partition:[2,3,1,""],max_partition:[2,3,1,""]},"airflow.models":{BaseOperator:[2,0,1,""],Connection:[2,0,1,""],DAG:[2,0,1,""],DagBag:[2,0,1,""],TaskInstance:[2,0,1,""]},"airflow.models.BaseOperator":{clear:[2,2,1,""],dag:[2,4,1,""],deps:[2,4,1,""],detect_downstream_cycle:[2,2,1,""],downstream_list:[2,4,1,""],execute:[2,2,1,""],get_direct_relatives:[2,2,1,""],get_flat_relatives:[2,2,1,""],get_task_instances:[2,2,1,""],has_dag:[2,2,1,""],on_kill:[2,2,1,""],post_execute:[2,2,1,""],pre_execute:[2,2,1,""],prepare_template:[2,2,1,""],render_template:[2,2,1,""],render_template_from_field:[2,2,1,""],run:[2,2,1,""],schedule_interval:[2,4,1,""],set_downstream:[2,2,1,""],set_upstream:[2,2,1,""],upstream_list:[2,4,1,""],xcom_pull:[2,2,1,""],xcom_push:[2,2,1,""]},"airflow.models.Connection":{extra_dejson:[2,4,1,""]},"airflow.models.DAG":{add_task:[2,2,1,""],add_task
s:[2,2,1,""],clear:[2,2,1,""],cli:[2,2,1,""],concurrency_reached:[2,4,1,""],crawl_for_tasks:[2,2,1,""],create_dagrun:[2,2,1,""],deactivate_stale_dags:[2,5,1,""],deactivate_unknown_dags:[2,5,1,""],filepath:[2,4,1,""],folder:[2,4,1,""],get_active_runs:[2,2,1,""],get_dagrun:[2,2,1,""],get_last_dagrun:[2,2,1,""],get_template_env:[2,2,1,""],is_paused:[2,4,1,""],latest_execution_date:[2,4,1,""],normalize_schedule:[2,2,1,""],run:[2,2,1,""],set_dependency:[2,2,1,""],sub_dag:[2,2,1,""],subdags:[2,4,1,""],sync_to_db:[2,5,1,""],tree_view:[2,2,1,""]},"airflow.models.DagBag":{bag_dag:[2,2,1,""],collect_dags:[2,2,1,""],dagbag_report:[2,2,1,""],get_dag:[2,2,1,""],kill_zombies:[2,2,1,""],process_file:[2,2,1,""],size:[2,2,1,""]},"airflow.models.TaskInstance":{are_dependencies_met:[2,2,1,""],are_dependents_done:[2,2,1,""],clear_xcom_data:[2,2,1,""],command:[2,2,1,""],command_as_list:[2,2,1,""],current_state:[2,2,1,""],error:[2,2,1,""],generate_command:[2,5,1,""],get_dagrun:[2,2,1,""],init_on_load:[2,
2,1,""],is_premature:[2,4,1,""],key:[2,4,1,""],next_retry_datetime:[2,2,1,""],pool_full:[2,2,1,""],previous_ti:[2,4,1,""],ready_for_retry:[2,2,1,""],refresh_from_db:[2,2,1,""],run:[2,2,1,""],xcom_pull:[2,2,1,""],xcom_push:[2,2,1,""]},"airflow.operators":{BashOperator:[2,0,1,""],BranchPythonOperator:[2,0,1,""],DummyOperator:[2,0,1,""],EmailOperator:[2,0,1,""],ExternalTaskSensor:[2,0,1,""],GenericTransfer:[2,0,1,""],HdfsSensor:[2,0,1,""],HivePartitionSensor:[2,0,1,""],HttpSensor:[2,0,1,""],MetastorePartitionSensor:[2,0,1,""],MySqlOperator:[2,0,1,""],NamedHivePartitionSensor:[2,0,1,""],PrestoCheckOperat
[2/2] incubator-airflow-site git commit: v1.8.0
v1.8.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo
Commit:
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/69cff492
Tree:
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/69cff492
Diff:
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/69cff492
Branch: refs/heads/asf-site
Commit: 69cff49228f32d88fcdf3e92808ad51ab3438d8d
Parents: 5e57401
Author: Maxime Beauchemin
Authored: Mon Mar 20 17:04:37 2017 -0700
Committer: Maxime Beauchemin
Committed: Mon Mar 20 17:04:37 2017 -0700
--
.../contrib/executors/mesos_executor.html | 2 +-
_modules/airflow/models.html| 134 ---
_modules/airflow/operators/sensors.html | 2 +-
_modules/mysql_hook.html| 15 +--
_modules/mysql_operator.html| 8 +-
_modules/sensors.html | 2 +-
_sources/concepts.rst.txt | 3 +-
_sources/configuration.rst.txt | 35 -
code.html | 27 ++--
concepts.html | 3 +-
configuration.html | 28
genindex.html | 2 +
index.html | 1 -
objects.inv | Bin 2147 -> 2159 bytes
searchindex.js | 2 +-
15 files changed, 123 insertions(+), 141 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/69cff492/_modules/airflow/contrib/executors/mesos_executor.html
--
diff --git a/_modules/airflow/contrib/executors/mesos_executor.html
b/_modules/airflow/contrib/executors/mesos_executor.html
index dbca37d..311d1a8 100644
--- a/_modules/airflow/contrib/executors/mesos_executor.html
+++ b/_modules/airflow/contrib/executors/mesos_executor.html
@@ -331,7 +331,7 @@
except KeyError:
# The map may not contain an item if the
framework re-registered after a failover.
# Discard these tasks.
-logging.warning("Unrecognised task key %s" % update.task_id.value)
+logging.warn("Unrecognised task key %s" % update.task_id.value)
return
if update.state == mesos_pb2.TASK_FINISHED:
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/69cff492/_modules/airflow/models.html
--
diff --git a/_modules/airflow/models.html b/_modules/airflow/models.html
index a94686b..0b043ea 100644
--- a/_modules/airflow/models.html
+++ b/_modules/airflow/models.html
@@ -632,7 +632,7 @@
def paused_dags(self):
session = settings.Session()
dag_ids = [dp.dag_id for dp
in session.query(DagModel).filter(
-DagModel.is_paused.__eq__(True))]
+DagModel.is_paused.is_(True))]
session.commit()
session.close()
return dag_ids
@@ -1161,6 +1161,7 @@
self.end_date = ti.end_date
self.try_number = ti.try_number
self.hostname = ti.hostname
+self.pid = ti.pid
else:
self.state = None
@@ -1452,19 +1453,20 @@
verbose=True)
if not runnable and not mark_success:
-if self.state != State.QUEUED:
-# If a task's dependencies are met but it
can't be run yet then queue it
-# instead
-self.state = State.QUEUED
-msg = "Queuing attempt {attempt} of {total}".format(
-attempt=self.try_number % (task.retries + 1) + 1,
-total=task.retries
+ 1)
-logging.info(hr + msg + hr)
-
-self.queued_dttm = datetime.now()
-msg = "Queuing into pool {}".format(self.pool)
-logging.info(msg)
-session.merge(self)
+# FIXME: we might have hit concurrency limits,
which means we probably
+# have been running prematurely. This should be
handled in the
+# scheduling mechanism.
+self.state = State.NONE
+msg = ("FIXME: Rescheduling due to concurrency
limits reached at task "
+ "runtime. Attempt {attempt} of {total}. State set to
NONE.").format(
+attempt=self.try_number
% (task.retries
+ 1)
+ 1,
+total=task.retries
+ 1)
+logging.warning(hr + msg + hr)
+
+self.queued_dttm = datetime.now()
+msg = "Queuing into pool {}".format(self.pool)
+logging.info(msg)
+session.merge(self)
session.commit()
return
@@ -
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.7.1.3-docs [created] 209bf9c7b
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.6.2-docs [created] 35cb43992
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.7.1.3 c643a553c -> 209bf9c7b
[incubator-airflow] Git Push Summary
Repository: incubator-airflow Updated Tags: refs/tags/1.7.0rc1 [created] ffc070100 refs/tags/1.7.1rc1 [created] 658185870 refs/tags/airbnb_1.7.1rc1 [created] 658185870 refs/tags/airbnb_1.7.1rc10 [created] 8d7297573 refs/tags/airbnb_1.7.1rc11 [created] 39ab764a9 refs/tags/airbnb_1.7.1rc13 [created] 15637b0ed refs/tags/airbnb_1.7.1rc14 [created] 061dfc6cc refs/tags/airbnb_1.7.1rc2 [created] 173b19313 refs/tags/airbnb_1.7.1rc3 [created] f1ff65c8c refs/tags/airbnb_1.7.1rc4 [created] 173b19313 refs/tags/airbnb_1.7.1rc6 [created] 563be1324 refs/tags/airbnb_1.7.1rc7 [created] f4e9fd68e refs/tags/airbnb_1.7.1rc8 [created] 51ee7d36c refs/tags/airbnb_prod.1.6.1.0 [created] 546e54cef refs/tags/airbnb_prod.1.6.1.1 [created] 190ee9398 refs/tags/airbnb_prod.1.6.1.2 [created] 3be5e8115 refs/tags/airbnb_prod.1.6.1.3 [created] 0f69d352e refs/tags/airbnb_prod.1.6.1.4 [created] a0e41d32e refs/tags/airbnb_prod.1.6.1.5 [created] 715b98509 refs/tags/airbnb_prod.1.6.1.6 [created] e19bc21ff refs/tags/airbnb_prod.1.6.1.7 [created] 101c95782 refs/tags/airbnb_prod.1.6.1.8 [created] 072f4e8d2 refs/tags/airbnb_prod.1.6.2.1 [created] 85ff87ee1 refs/tags/airbnb_prod.1.6.2.10 [created] ea16895a0 refs/tags/airbnb_prod.1.6.2.2 [created] d233e41da refs/tags/airbnb_prod.1.6.2.3 [created] bd0966069 refs/tags/airbnb_prod.1.6.2.4 [created] 2c150966f refs/tags/airbnb_prod.1.6.2.5 [created] 0f5ff784d refs/tags/airbnb_prod.1.6.2.6 [created] 95fa58fbd refs/tags/airbnb_prod.1.6.2.7 [created] f74e6900e refs/tags/airbnb_prod.1.6.2.8 [created] 7866d5d32 refs/tags/airbnb_prod.1.6.2.9 [created] ffc070100
[2/2] incubator-airflow git commit: Merge pull request #2130 from seancron/airflow-954
Merge pull request #2130 from seancron/airflow-954 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/abbb4ee5 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/abbb4ee5 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/abbb4ee5 Branch: refs/heads/master Commit: abbb4ee5cd02c691c611631a79212806f77c0fb2 Parents: d7f8334 f392469 Author: Maxime Beauchemin Authored: Wed Mar 8 21:08:00 2017 -0800 Committer: Maxime Beauchemin Committed: Wed Mar 8 21:08:00 2017 -0800 -- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) --
[1/2] incubator-airflow git commit: [AIRFLOW-954] Fix configparser ImportError
Repository: incubator-airflow Updated Branches: refs/heads/master d7f8334d2 -> abbb4ee5c [AIRFLOW-954] Fix configparser ImportError Fixes support for Python 2.7 since https://github.com/apache/incubator-airflow/pull/2091 was merged Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f3924696 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f3924696 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f3924696 Branch: refs/heads/master Commit: f3924696ff32942abc8c5709876c5717e0a82445 Parents: e79dee8 Author: Sean Cronin Authored: Tue Mar 7 20:52:48 2017 -0500 Committer: Sean Cronin Committed: Tue Mar 7 20:52:48 2017 -0500 -- setup.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f3924696/setup.py -- diff --git a/setup.py b/setup.py index 87808d6..481d427 100644 --- a/setup.py +++ b/setup.py @@ -202,6 +202,7 @@ def do_setup(): scripts=['airflow/bin/airflow'], install_requires=[ 'alembic>=0.8.3, <0.9', +'configparser>=3.5.0, <3.6.0', 'croniter>=0.3.8, <0.4', 'dill>=0.2.2, <0.3', 'flask>=0.11, <0.12', @@ -211,7 +212,7 @@ def do_setup(): 'flask-swagger==0.2.13', 'flask-wtf==0.12', 'funcsigs==1.0.0', -'future>=0.15.0, <0.17', +'future>=0.16.0, <0.17', 'gitpython>=2.0.2', 'gunicorn>=19.3.0, <19.4.0', # 19.4.? seemed to have issues 'jinja2>=2.7.3, <2.9.0',
[2/2] incubator-airflow git commit: Merge pull request #2133 from dhuang/AIRFLOW-956
Merge pull request #2133 from dhuang/AIRFLOW-956 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/d7f8334d Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/d7f8334d Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/d7f8334d Branch: refs/heads/master Commit: d7f8334d246bd3c6aa374fdfdb46fb59bb6bb626 Parents: e79dee8 f4be54f Author: Maxime Beauchemin Authored: Wed Mar 8 20:56:41 2017 -0800 Committer: Maxime Beauchemin Committed: Wed Mar 8 20:56:41 2017 -0800 -- .readthedocs.yml | 20 1 file changed, 20 insertions(+) --
[1/2] incubator-airflow git commit: [AIRFLOW-956] Get docs working on readthedocs.org
Repository: incubator-airflow Updated Branches: refs/heads/master e79dee871 -> d7f8334d2 [AIRFLOW-956] Get docs working on readthedocs.org Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/f4be54f2 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/f4be54f2 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/f4be54f2 Branch: refs/heads/master Commit: f4be54f2535ca13819e20e742a8ed6ae522d1a17 Parents: e79dee8 Author: Daniel Huang Authored: Wed Mar 8 19:42:29 2017 -0800 Committer: Daniel Huang Committed: Wed Mar 8 19:42:29 2017 -0800 -- .readthedocs.yml | 20 1 file changed, 20 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/f4be54f2/.readthedocs.yml -- diff --git a/.readthedocs.yml b/.readthedocs.yml new file mode 100644 index 000..ca5b56d --- /dev/null +++ b/.readthedocs.yml @@ -0,0 +1,20 @@ +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +python: +pip_install: true +extra_requirements: +- doc +- docker +- gcp_api +- emr
[10/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/down-pressed.png -- diff --git a/_static/down-pressed.png b/_static/down-pressed.png index 7c30d00..5756c8c 100644 Binary files a/_static/down-pressed.png and b/_static/down-pressed.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/down.png -- diff --git a/_static/down.png b/_static/down.png index f48098a..1b3bdad 100644 Binary files a/_static/down.png and b/_static/down.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/file.png -- diff --git a/_static/file.png b/_static/file.png index 254c60b..a858a41 100644 Binary files a/_static/file.png and b/_static/file.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Inconsolata-Bold.ttf -- diff --git a/_static/fonts/Inconsolata-Bold.ttf b/_static/fonts/Inconsolata-Bold.ttf index 58c9fef..809c1f5 100644 Binary files a/_static/fonts/Inconsolata-Bold.ttf and b/_static/fonts/Inconsolata-Bold.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Inconsolata-Regular.ttf -- diff --git a/_static/fonts/Inconsolata-Regular.ttf b/_static/fonts/Inconsolata-Regular.ttf index a87ffba..fc981ce 100644 Binary files a/_static/fonts/Inconsolata-Regular.ttf and b/_static/fonts/Inconsolata-Regular.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Lato-Bold.ttf -- diff --git a/_static/fonts/Lato-Bold.ttf b/_static/fonts/Lato-Bold.ttf index 7434369..1d23c70 100644 Binary files a/_static/fonts/Lato-Bold.ttf and b/_static/fonts/Lato-Bold.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Lato-Regular.ttf -- diff --git a/_static/fonts/Lato-Regular.ttf b/_static/fonts/Lato-Regular.ttf index 04ea8ef..0f3d0f8 100644 Binary files a/_static/fonts/Lato-Regular.ttf and b/_static/fonts/Lato-Regular.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.eot -- diff --git a/_static/fonts/fontawesome-webfont.eot b/_static/fonts/fontawesome-webfont.eot index 84677bc..c7b00d2 100644 Binary files a/_static/fonts/fontawesome-webfont.eot and b/_static/fonts/fontawesome-webfont.eot differ
[19/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/api.html -- diff --git a/api.html b/api.html new file mode 100644 index 000..7aca3bb --- /dev/null +++ b/api.html @@ -0,0 +1,279 @@ + + + + + + + + + + + Experimental Rest API — Airflow Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Airflow + + + + + + + + + + + + + + + + + + + + + + + + + + + +Project +License +Quick Start +Installation +Tutorial +Configuration +UI / Screenshots +Concepts +Data Profiling +Command Line Interface +Scheduling & Triggers +Plugins +Security +Experimental Rest API +Endpoints +CLI +Authentication + + +Integration +FAQ +API Reference + + + + + + + + + + + + + + + Airflow + + + + + + + + + + + + + + + + + + + + + + + + + + + + Docs » + + Experimental Rest API + + + + + + View page source + + + + + + + + + + http://schema.org/Article";> + + + +Experimental Rest API¶ +Airflow exposes an experimental Rest API. It is available through the webserver. Endpoints are +available at /api/experimental/. Please note that we expect the endpoint definitions to change. + +Endpoints¶ +This is a place holder until the swagger definitions are active + +/api/experimental/dags//tasks/ returns info for a task (GET). +/api/experimental/dags/ /dag_runs creates a dag_run for a given dag id (POST). + + + +CLI¶ +For some functions the cli can use the API. To configure the CLI to use the API when available +configure as follows: +[cli] +api_client = airflow.api.client.json_client +endpoint_url = http:// : ; + + + + +Authentication¶ +Only Kerberos authentication is currently supported for the API. To enable this set the following +in the configuration: +[api] +auth_backend = airflow.api.auth.backend.default + +[kerberos] +keytab = + + +The Kerberos service is configured as airflow/fully.qualified.domainname@REALM. Make sure this +principal exists in the keytab file. + + + + + + + + + + + + + +Next + + + Previous + + + + + + + + + + + + Built with http://sphinx-doc.org/";>Sphinx using a https://github.com/snide/sphinx_rtd_theme";>theme provided by https://readthedocs.org";>Read the Docs. + + + + + + + + + + + + + + + +var DOCUMENTATION_OPTIONS = { +URL_ROOT:'./', +VERSION:'', +COLLAPSE_INDEX:false, +FILE_SUFFIX:'.html', +HAS_SOURCE: true, +SOURCELINK_SUFFIX: '.txt' +}; + + + + + + + + + + + + + + + + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/integration.html -- diff --git a/integration.html b/integration.html new file mode 100644 index 000..a55aa41 --- /dev/null +++ b/integration.html @@ -0,0 +1,424 @@ + + + + + + + + + + + Integration — Airflow Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Airflow + + + + + + + + + + + + + + + + + + + + + + + + + + + +Project +License +Quick Start +Installation +Tutorial +Configuration +UI / Screenshots +Concepts +Data Profiling +Command Line Interface +Scheduling & Triggers +Plugins +Security +Experimental Rest API +Integration +AWS: Amazon Webservices +GCP: Google Cloud Platform +BigQuery +BigQuery Operators +BigQueryHook + + +Cloud DataFlow +DataFlow Operators +DataFlowHook + + +Cloud DataProc +DataProc Operators +DataProcPySparkOperator + + +Cloud Datastore +Datastore Operators + + +Cloud Storage +Storage Operators +GoogleCloudStorageHook + + + + + + +FAQ +API Reference + + + + + + + + + + + +
[18/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
Latest docs version as of 1.8.x Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/9c75ee9e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/9c75ee9e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/9c75ee9e Branch: refs/heads/asf-site Commit: 9c75ee9e46cc5ec459979dbfd2c911743c07fd63 Parents: 4af0850 Author: Maxime Beauchemin Authored: Mon Mar 6 08:43:12 2017 -0800 Committer: Maxime Beauchemin Committed: Mon Mar 6 08:43:12 2017 -0800 -- .../contrib/executors/mesos_executor.html | 81 +- .../contrib/operators/hipchat_operator.html | 72 +- _modules/airflow/executors/local_executor.html | 73 +- .../airflow/executors/sequential_executor.html | 59 +- _modules/airflow/macros.html| 79 +- _modules/airflow/macros/hive.html | 61 +- _modules/airflow/models.html| 1993 ++ _modules/airflow/operators/sensors.html | 242 ++- _modules/bash_operator.html | 77 +- _modules/dagrun_operator.html | 67 +- _modules/dbapi_hook.html| 130 +- _modules/dummy_operator.html| 49 +- _modules/email_operator.html| 61 +- _modules/ftp_hook.html | 69 +- _modules/generic_transfer.html | 57 +- _modules/http_hook.html | 61 +- _modules/http_operator.html | 71 +- _modules/index.html | 74 +- _modules/mysql_hook.html| 95 +- _modules/mysql_operator.html| 61 +- _modules/presto_check_operator.html | 60 +- _modules/presto_hook.html | 63 +- _modules/python_operator.html | 69 +- _modules/sensors.html | 242 ++- _modules/sqlite_hook.html | 49 +- _modules/ssh_execute_operator.html | 75 +- _modules/ssh_hook.html | 81 +- _static/basic.css | 68 +- _static/comment-bright.png | Bin 3500 -> 756 bytes _static/comment-close.png | Bin 3578 -> 829 bytes _static/comment.png | Bin 3445 -> 641 bytes _static/css/badge_only.css |2 +- _static/css/theme.css |4 +- _static/down-pressed.png| Bin 347 -> 222 bytes _static/down.png| Bin 347 -> 202 bytes _static/file.png| Bin 358 -> 286 bytes _static/fonts/Inconsolata-Bold.ttf | Bin 66352 -> 109948 bytes _static/fonts/Inconsolata-Regular.ttf | Bin 84548 -> 96964 bytes _static/fonts/Lato-Bold.ttf | Bin 121788 -> 656544 bytes _static/fonts/Lato-Regular.ttf | Bin 120196 -> 656568 bytes _static/fonts/fontawesome-webfont.eot | Bin 56006 -> 76518 bytes _static/fonts/fontawesome-webfont.svg | 207 +- _static/fonts/fontawesome-webfont.ttf | Bin 112160 -> 152796 bytes _static/fonts/fontawesome-webfont.woff | Bin 65452 -> 90412 bytes _static/jquery.js |8 +- _static/js/theme.js | 58 +- _static/minus.png | Bin 173 -> 90 bytes _static/plus.png| Bin 173 -> 90 bytes _static/searchtools.js | 115 +- _static/up-pressed.png | Bin 345 -> 214 bytes _static/up.png | Bin 345 -> 203 bytes cli.html| 482 +++-- code.html | 1933 +++-- concepts.html | 129 +- configuration.html | 97 +- faq.html| 61 +- genindex.html | 1541 -- index.html | 115 +- installation.html | 51 +- license.html| 51 +- objects.inv | Bin 2326 -> 2147 bytes plugins.html| 64 +- profiling.html | 51 +- project.html| 76 +- py-modindex.html| 63 +- scheduler.html | 79 +- search.html | 41 +- searchindex.js
[16/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/airflow/operators/sensors.html
--
diff --git a/_modules/airflow/operators/sensors.html
b/_modules/airflow/operators/sensors.html
index 3b716d3..6d51369 100644
--- a/_modules/airflow/operators/sensors.html
+++ b/_modules/airflow/operators/sensors.html
@@ -30,6 +30,9 @@
+
+
@@ -40,6 +43,7 @@
+
@@ -90,6 +94,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API Reference
@@ -104,8 +110,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -118,19 +126,36 @@
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
+
+ Docs »
+
Module code »
-
-airflow.operators.sensors
+
+ airflow.operators.sensors
+
+
-
+
+
+
+
http://schema.org/Article";>
@@ -151,24 +176,27 @@
# See the License for the specific language governing
permissions and
# limitations under the License.
-from __future__ import print_function
-from future import standard_library
+from __future__ import print_function
+from future import standard_library
standard_library.install_aliases()
-from builtins import str
-from past.builtins import basestring
+from builtins import str
+from past.builtins import basestring
-from datetime import datetime
+from datetime import datetime
import logging
-from urllib.parse import urlparse
-from time import sleep
+from urllib.parse import urlparse
+from time import sleep
+import re
+import sys
import airflow
-from airflow import hooks,
settings
-from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException
-from airflow.models import BaseOperator, TaskInstance,
Connection as DB
-from airflow.hooks.base_hook
import BaseHook
-from airflow.utils.state import State
-from airflow.utils.decorators
import apply_defaults
+from airflow import hooks,
settings
+from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException
+from airflow.models import BaseOperator, TaskInstance
+from airflow.hooks.base_hook
import BaseHook
+from airflow.hooks.hdfs_hook
import HDFSHook
+from airflow.utils.state import State
+from airflow.utils.decorators
import apply_defaults
[docs]class BaseSensorOperator(BaseOperator):
@@ -193,7 +221,7 @@
self,
poke_interval=60,
timeout=60*60*24*7,
-soft_fail=False,
+soft_fail=False,
*args, **kwargs):
super(BaseSensorOperator, self).__init__(*args, **kwargs)
self.poke_interval = poke_interval
@@ -245,13 +273,13 @@
logging.info('Poking:
' + self.sql)
records = hook.get_records(self.sql)
if not records:
-return False
+return False
else:
if str(records[0][0]) in ('0', '',):
-return False
+return False
else:
-return True
-print(records[0][0])
+return True
+print(records[0][0])
class MetastorePartitionSensor(SqlSensor):
@@ -286,13 +314,18 @@
self.partition_name = partition_name
self.table = table
self.schema = schema
-self.first_poke = True
+self.first_poke = True
self.conn_id = mysql_conn_id
+# TODO(aoen): We shouldn't be using SqlSensor
here but MetastorePartitionSensor.
+# The problem is the way apply_defaults works
isn't compatible with inheritance.
+# The inheritance model needs to be reworked in order
to support overriding args/
+# kwargs with arguments here, then 'conn_id'
and 'sql' can be passed into the
+# constructor below and apply_defaults will no longer
throw an exception.
super(SqlSensor, self).__init__(*args, **kwargs)
def poke(self, context):
if self.first_poke:
-self.first_poke = False
+self.first_poke = False
if '.'
in self.table:
self.schema, self.table
= self.table.split('.')
self.sql = """
@@ -301,9 +334,9 @@
LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
B0.TBL_ID
LEFT OUTER JOIN DBS C0 ON B0.DB_ID =
C0.DB_ID
WHERE
-B0.TBL_NAME = '{self.table}'
AND
-C0.NAME = '{self.schema}' AND
-A0.PART_NAME =
'{self.partition_name}';
+B0.TBL_NAME = '{self.table}' AND
+C0.NAME = '{self.schema}' AND
+A0.PART_NAME = '{self.partition_name}';
""".format(self=self)
return super(MetastorePartitionSensor, self).poke(context)
@@ -338,13 +371,13 @@
self,
[02/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/searchindex.js
--
diff --git a/searchindex.js b/searchindex.js
index 3d2f62d..625cb26 100644
--- a/searchindex.js
+++ b/searchindex.js
@@ -1 +1 @@
-Search.setIndex({envversion:47,filenames:["cli","code","concepts","configuration","faq","index","installation","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],objects:{"airflow.contrib":{hooks:[1,1,0,"-"],operators:[1,1,0,"-"]},"airflow.contrib.executors.mesos_executor":{MesosExecutor:[1,0,1,""]},"airflow.contrib.hooks":{BigQueryHook:[1,0,1,""],CloudantHook:[1,0,1,""],FTPHook:[1,0,1,""],GoogleCloudStorageHook:[1,0,1,""],SSHHook:[1,0,1,""],VerticaHook:[1,0,1,""]},"airflow.contrib.hooks.BigQueryHook":{get_conn:[1,2,1,""],get_pandas_df:[1,2,1,""],get_service:[1,2,1,""],insert_rows:[1,2,1,""]},"airflow.contrib.hooks.CloudantHook":{db:[1,2,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[1,2,1,""],create_directory:[1,2,1,""],delete_directory:[1,2,1,""],delete_file:[1,2,1,""],describe_directory:[1,2,1,""],get_conn:[1,2,1,""],list_directory:[1,2,1,""],rename:[1,2,1,""],retrieve_file:[1,2,1,""],store_file:[1,2,1,""]},"airflow.contrib.hooks.Google
CloudStorageHook":{download:[1,2,1,""],get_conn:[1,2,1,""],upload:[1,2,1,""]},"airflow.contrib.hooks.SSHHook":{Popen:[1,2,1,""],check_output:[1,2,1,""],tunnel:[1,2,1,""]},"airflow.contrib.hooks.VerticaHook":{get_conn:[1,2,1,""]},"airflow.contrib.hooks.gcs_hook":{GoogleCloudStorageHook:[1,0,1,""]},"airflow.contrib.operators":{QuboleOperator:[1,0,1,""],SSHExecuteOperator:[1,0,1,""],VerticaOperator:[1,0,1,""],VerticaToHiveTransfer:[1,0,1,""]},"airflow.contrib.operators.bigquery_operator":{BigQueryOperator:[1,0,1,""]},"airflow.contrib.operators.bigquery_to_gcs":{BigQueryToCloudStorageOperator:[1,0,1,""]},"airflow.contrib.operators.gcs_download_operator":{GoogleCloudStorageDownloadOperator:[1,0,1,""]},"airflow.contrib.operators.hipchat_operator":{HipChatAPIOperator:[1,0,1,""],HipChatAPISendRoomNotificationOperator:[1,0,1,""]},"airflow.executors":{CeleryExecutor:[1,0,1,""],LocalExecutor:[1,0,1,""],SequentialExecutor:[1,0,1,""]},"airflow.hooks":{DbApiHook:[1,0,1,""],DruidHook:[1,0,1,""],Hi
veCliHook:[1,0,1,""],HiveMetastoreHook:[1,0,1,""],HiveServer2Hook:[1,0,1,""],HttpHook:[1,0,1,""],MsSqlHook:[1,0,1,""],MySqlHook:[1,0,1,""],PostgresHook:[1,0,1,""],PrestoHook:[1,0,1,""],S3Hook:[1,0,1,""],SqliteHook:[1,0,1,""],WebHDFSHook:[1,0,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[1,2,1,""],bulk_load:[1,2,1,""],get_conn:[1,2,1,""],get_cursor:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],insert_rows:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.DruidHook":{construct_ingest_query:[1,2,1,""],get_conn:[1,2,1,""],load_from_hdfs:[1,2,1,""]},"airflow.hooks.HiveCliHook":{load_file:[1,2,1,""],run_cli:[1,2,1,""],test_hql:[1,2,1,""]},"airflow.hooks.HiveMetastoreHook":{check_for_named_partition:[1,2,1,""],check_for_partition:[1,2,1,""],get_databases:[1,2,1,""],get_metastore_client:[1,2,1,""],get_partitions:[1,2,1,""],get_table:[1,2,1,""],get_tables:[1,2,1,""],max_partition:[1,2,1,""],table_exists:[1,2,1,""]},"airflow.hooks.HiveServer2Hook":{get_pandas_df:[1,2,
1,""],get_records:[1,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[1,2,1,""],run:[1,2,1,""],run_and_check:[1,2,1,""]},"airflow.hooks.MsSqlHook":{get_conn:[1,2,1,""]},"airflow.hooks.MySqlHook":{bulk_load:[1,2,1,""],get_conn:[1,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.S3Hook":{check_for_bucket:[1,2,1,""],check_for_key:[1,2,1,""],check_for_prefix:[1,2,1,""],check_for_wildcard_key:[1,2,1,""],get_bucket:[1,2,1,""],get_conn:[1,2,1,""],get_key:[1,2,1,""],get_wildcard_key:[1,2,1,""],list_keys:[1,2,1,""],list_prefixes:[1,2,1,""],load_file:[1,2,1,""],load_string:[1,2,1,""]},"airflow.hooks.SqliteHook":{get_conn:[1,2,1,""]},"airflow.hooks.WebHDFSHook":{check_for_path:[1,2,1,""],get_conn:[1,2,1,""],load_file:[1,2,1,""]},"airflow.macros":{ds_add:[1,3,1,""],ds_format:[1,3,1,""],hive:[1,1,0,"-"],random:[1,3,1,""]},"airflow.macros.hive":{closest_ds_partition:[1,3,1,""],max_partition:[1,
3,1,""]},"airflow.models":{BaseOperator:[1,0,1,""],Connection:[1,0,1,""],DAG:[1,0,1,""],DagBag:[1,0,1,""],TaskInstance:[1,0,1,""]},"airflow.models.BaseOperator":{clear:[1,2,1,""],dag:[1,4,1,""],detect_downstream_cycle:[1,2,1,""],downstream_list:[1,4,1,""],execute:[1,2,1,""],get_direct_relatives:[1,2,1,""],get_flat_relatives:[1,2,1,""],get_task_instances:[1,2,1,""],has_dag:[1,2,1,""],on_kill:[1,2,1,""],post_execute:[1,2,1,""],pre_execute:[1,2,1,""],prepare_template:[1,2,1,""],render_template:[1,2,1,""],render_template_from_field:[1,2,1,""],run:[1,2,1,""],schedule_interval:[1,4,1,""],set_downstream:[1,2,1,""],set_upstream:[1,2,1,""],upstream_list:[1,4,1,""],xcom_pull:[1,2,1,""],xcom_push:[1,2,1,""]},
[01/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
Repository: incubator-airflow-site Updated Branches: refs/heads/asf-site 4af0850c3 -> 5e5740122 http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/security.html -- diff --git a/security.html b/security.html index 25ada3c..d0156b7 100644 --- a/security.html +++ b/security.html @@ -30,8 +30,11 @@ + + - + @@ -41,6 +44,7 @@ + @@ -102,12 +106,27 @@ Limitations Enabling kerberos Using kerberos authentication -GitHub Enterprise (GHE) Authentication -Setting up GHE Authentication +OAuth Authentication +GitHub Enterprise (GHE) Authentication +Setting up GHE Authentication + + +Google Authentication +Setting up Google Authentication + + + + +SSL +Impersonation + + +Experimental Rest API +Integration FAQ API Reference @@ -122,8 +141,10 @@ - -Airflow + + + Airflow + @@ -132,23 +153,40 @@ - + + + + + + + + + + + + -Docs » - -Security + + Docs » + + Security + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -156,13 +194,13 @@ Security¶ - -Web Authentication¶ By default, all gates are opened. An easy way to restrict access to the web application is to do it at the network level, or by using SSH tunnels. It is however possible to switch on authentication by either using one of the supplied backends or create your own. + +Web Authentication¶ Password¶ One of the simplest mechanisms for authentication is requiring users to specify a password before logging in. @@ -331,6 +369,9 @@ section of the connection. For the login user specify the following as extra: + + +OAuth Authentication¶ GitHub Enterprise (GHE) Authentication¶ The GitHub Enterprise authentication backend can be used to authenticate users @@ -348,12 +389,11 @@ your GHE installation will be able to login to Airflow. client_id = oauth_key_from_github_enterprise client_secret = oauth_secret_from_github_enterprise oauth_callback_route = /example/ghe_oauth/callback -allowed_teams = example_team_1, example_team_2 +allowed_teams = 1, 345, 23 - -Setting up GHE Authentication¶ +Setting up GHE Authentication¶ An application must be setup in GHE before you can use the GHE authentication backend. In order to setup an application: @@ -367,19 +407,87 @@ backend. In order to setup an application: + +Google Authentication¶ +The Google authentication backend can be used to authenticate users +against Google using OAuth2. You must specify a domain to restrict login +to only members of that domain. +[webserver] +authenticate = True +auth_backend = airflow.contrib.auth.backends.google_auth + +[google] +client_id = google_client_id +client_secret = google_client_secret +oauth_callback_route = /oauth2callback +domain = example.com + + + +Setting up Google Authentication¶ +An application must be setup in the Google API Console before you can use the Google authentication +backend. In order to setup an application: + +Navigate to https://console.developers.google.com/apis/";>https://console.developers.google.com/apis/ +Select ‘Credentials’ from the left hand nav +Click ‘Create credentials’ and choose ‘OAuth client ID’ +Choose ‘Web application’ +Fill in the required information (the ‘Authorized redirect URIs’ must be fully qualifed e.g. http://airflow.example.com/oauth2callback";>http://airflow.example.com/oauth2callback) +Click ‘Create’ +Copy ‘Client ID’, ‘Client Secret’, and your redirect URI to your airflow.cfg according to the above example + + + + + +SSL¶ +SSL can be enabled by providing a certificate and key. Once enabled, be sure to use +“https://";>https://” in your browser. +[webserver] +web_server_ssl_cert =+web_server_ssl_key = + + +Enabling SSL will not automatically change the web server port. If you want to use the +standard port 443, you’ll need to configure that too. Be aware that super user privileges +(or cap_net_bind_service on Linux) are required to listen on port 443. +# Optionally, set the server to listen on the standard SSL port. +web_server_port = 443 +base_url = http:// :443 + + + +Impersonation¶ +Airflow has the ability to impersonate a unix user while running task +instances based on the task’s run_as_user parameter, which takes a user’s name. +NOTE For impersonations to work, Airflow must be run with sudo as subtasks are run +with sudo -u and permissions of files are changed. Furthermore, the unix user needs to +exist on the worker. Here is what a simple sudoers file entry could look like to achieve +this, ass
[22/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
Latest docs version as of 1.8.x Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/5e574012 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/5e574012 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/5e574012 Branch: refs/heads/asf-site Commit: 5e5740122ed33a22a30047e75e6ca4c7da3961b4 Parents: 9c75ee9 Author: Maxime Beauchemin Authored: Mon Mar 6 08:43:25 2017 -0800 Committer: Maxime Beauchemin Committed: Mon Mar 6 08:43:25 2017 -0800 -- _images/latest_only_with_trigger.png | Bin 0 -> 40034 bytes _sources/api.rst.txt |43 + _sources/cli.rst.txt |11 + _sources/code.rst.txt| 255 + _sources/concepts.rst.txt| 833 +++ _sources/configuration.rst.txt | 284 + _sources/faq.rst.txt | 147 + _sources/index.rst.txt |89 + _sources/installation.rst.txt|90 + _sources/integration.rst.txt | 246 + _sources/license.rst.txt | 211 + _sources/plugins.rst.txt | 144 + _sources/profiling.rst.txt |39 + _sources/project.rst.txt |49 + _sources/scheduler.rst.txt | 153 + _sources/security.rst.txt| 334 + _sources/start.rst.txt |49 + _sources/tutorial.rst.txt| 429 ++ _sources/ui.rst.txt | 102 + _static/fonts/Inconsolata.ttf| Bin 0 -> 63184 bytes _static/jquery-3.1.0.js | 10074 api.html | 279 + integration.html | 424 ++ 23 files changed, 14285 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_images/latest_only_with_trigger.png -- diff --git a/_images/latest_only_with_trigger.png b/_images/latest_only_with_trigger.png new file mode 100644 index 000..629adfa Binary files /dev/null and b/_images/latest_only_with_trigger.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/api.rst.txt -- diff --git a/_sources/api.rst.txt b/_sources/api.rst.txt new file mode 100644 index 000..eef671c --- /dev/null +++ b/_sources/api.rst.txt @@ -0,0 +1,43 @@ +Experimental Rest API += + +Airflow exposes an experimental Rest API. It is available through the webserver. Endpoints are +available at /api/experimental/. Please note that we expect the endpoint definitions to change. + +Endpoints +- + +This is a place holder until the swagger definitions are active + +* /api/experimental/dags//tasks/ returns info for a task (GET). +* /api/experimental/dags//dag_runs creates a dag_run for a given dag id (POST). + +CLI +- + +For some functions the cli can use the API. To configure the CLI to use the API when available +configure as follows: + +.. code-block:: bash + +[cli] +api_client = airflow.api.client.json_client +endpoint_url = http://: + + +Authentication +-- + +Only Kerberos authentication is currently supported for the API. To enable this set the following +in the configuration: + +.. code-block:: bash + +[api] +auth_backend = airflow.api.auth.backend.default + +[kerberos] +keytab = + +The Kerberos service is configured as `airflow/fully.qualified.domainname@REALM`. Make sure this +principal exists in the keytab file. http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/cli.rst.txt -- diff --git a/_sources/cli.rst.txt b/_sources/cli.rst.txt new file mode 100644 index 000..f05cbfb --- /dev/null +++ b/_sources/cli.rst.txt @@ -0,0 +1,11 @@ +Command Line Interface +== + +Airflow has a very rich command line interface that allows for +many types of operation on a DAG, starting services, and supporting +development and testing. + +.. argparse:: + :module: airflow.bin.cli + :func: get_parser + :prog: airflow http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/code.rst.txt -- diff --git a/_sources/code.rst.txt b/_sources/code.rst.txt new file mode 100644 index 000..fabe6db --- /dev/null +++ b/_sources/code.rst.txt @@ -0,0 +1,255 @@ +API Reference += + +Operators +- +Operators allow for generation of certain types of tasks that become nodes in +the DAG when instantiated. All operators derive from BaseOperator and +inherit many at
[21/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/license.rst.txt -- diff --git a/_sources/license.rst.txt b/_sources/license.rst.txt new file mode 100644 index 000..9da26c0 --- /dev/null +++ b/_sources/license.rst.txt @@ -0,0 +1,211 @@ +License +=== + +.. image:: img/apache.jpg +:width: 150 + +:: + +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to
[05/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/concepts.html
--
diff --git a/concepts.html b/concepts.html
index 15b2b4d..1ed51a5 100644
--- a/concepts.html
+++ b/concepts.html
@@ -30,6 +30,9 @@
+
+
@@ -41,6 +44,7 @@
+
@@ -114,6 +118,7 @@
SubDAGs
SLAs
Trigger Rules
+Latest Run Only
Zombies & Undeads
Cluster Policy
Documentation & Notes
@@ -128,6 +133,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -142,8 +149,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -152,23 +161,40 @@
-
+
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
-Concepts
+
+ Docs »
+
+ Concepts
+
+
-
- View page
source
+
+ View page
source
+
+
+
http://schema.org/Article";>
@@ -341,8 +367,7 @@ object is always returned. For example:
We can put this all together to build a simple pipeline:
with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag:
(
-dag
->> DummyOperator(task_id='dummy_1')
+DummyOperator(task_id='dummy_1')
>> BashOperator(
task_id='bash_1',
bash_command='echo "HELLO!"')
@@ -452,10 +477,11 @@ variables from the operating system. The environment
variable needs to be
prefixed with AIRFLOW_CONN_ to be considered a connection. When
referencing the connection in the Airflow pipeline, the conn_id should
be the name of the variable without the prefix. For example, if the conn_id
-is named POSTGRES_MASTER the environment variable should be
named
-AIRFLOW_CONN_POSTGRES_MASTER. Airflow assumes the
value returned
-from the environment variable to be in a URI format
-(e.g. postgres://user:password@localhost:5432/master).
+is named postgres_master the environment variable should be
named
+AIRFLOW_CONN_POSTGRES_MASTER (note that the
environment variable must be
+all uppercase). Airflow assumes the value returned from the environment
+variable to be in a URI format (e.g.
+postgres://user:password@localhost:5432/master or
s3://accesskey:secretkey@S3).
Queues¶
@@ -674,6 +700,71 @@ while creating tasks:
that, when set to True, keeps a task from getting triggered if the
previous schedule for the task hasn’t succeeded.
+
+Latest Run Only¶
+Standard workflow behavior involves running a series of tasks for a
+particular date/time range. Some workflows, however, perform tasks that
+are independent of run time but need to be run on a schedule, much like a
+standard cron job. In these cases, backfills or running jobs missed during
+a pause just wastes CPU cycles.
+For situations like this, you can use the LatestOnlyOperator to skip
+tasks that are not being run during the most recent scheduled run for a
+DAG. The LatestOnlyOperator skips all immediate downstream
tasks, and
+itself, if the time right now is not between its execution_time and the
+next scheduled execution_time.
+One must be aware of the interaction between skipped tasks and trigger
+rules. Skipped tasks will cascade through trigger rules all_success
+and all_failed
but not all_done, one_failed, one_success,
+and dummy. If
you would like to use the LatestOnlyOperator with
+trigger rules that do not cascade skips, you will need to ensure that the
+LatestOnlyOperator is directly
upstream of the task you would like
+to skip.
+It is possible, through use of trigger rules to mix tasks that should run
+in the typical date/time dependent mode and those using the
+LatestOnlyOperator.
+For example, consider the following dag:
+#dags/latest_only_with_trigger.py
+import datetime as dt
+
+from airflow.models import DAG
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.operators.latest_only_operator import LatestOnlyOperator
+from airflow.utils.trigger_rule import
TriggerRule
+
+
+dag = DAG(
+dag_id='latest_only_with_trigger',
+schedule_interval=dt.timedelta(hours=4),
+start_date=dt.datetime(2016, 9, 20),
+)
+
+latest_only = LatestOnlyOperator(task_id='latest_only', dag=dag)
+
+task1 = DummyOperator(task_id='task1', dag=dag)
+task1.set_upstream(latest_only)
+
+task2 = DummyOperator(task_id='task2', dag=dag)
+
+task3 = DummyOperator(task_id='task3', dag=dag)
+task3.set_upstream([task1, task2])
+
+task4 = DummyOperator(task_id='task4', dag=dag,
+ trigger_rule=TriggerRule.ALL_DONE)
+task4.set_upstream([task1, task2])
+
+
+In the case of this dag, the latest_only task will show up as skipped
+for all runs except the latest run. task1 is directly downstream of
+latest_only and
will also skip for all runs except the latest.
+task2 is
entirely independent of latest_only an
[13/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/sensors.html
--
diff --git a/_modules/sensors.html b/_modules/sensors.html
index 6567e38..f2b1cfe 100644
--- a/_modules/sensors.html
+++ b/_modules/sensors.html
@@ -30,6 +30,9 @@
+
+
@@ -40,6 +43,7 @@
+
@@ -90,6 +94,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -104,8 +110,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -118,19 +126,36 @@
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
+
+ Docs »
+
Module code »
-
-sensors
+
+ sensors
+
+
-
+
+
+
+
http://schema.org/Article";>
@@ -151,24 +176,27 @@
# See the License for the specific language governing
permissions and
# limitations under the License.
-from __future__ import print_function
-from future import standard_library
+from __future__ import print_function
+from future import standard_library
standard_library.install_aliases()
-from builtins import str
-from past.builtins import basestring
+from builtins import str
+from past.builtins import basestring
-from datetime import datetime
+from datetime import datetime
import logging
-from urllib.parse import urlparse
-from time import sleep
+from urllib.parse import urlparse
+from time import sleep
+import re
+import sys
import airflow
-from airflow import hooks,
settings
-from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException
-from airflow.models import BaseOperator, TaskInstance,
Connection as DB
-from airflow.hooks.base_hook
import BaseHook
-from airflow.utils.state import State
-from airflow.utils.decorators
import apply_defaults
+from airflow import hooks,
settings
+from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException
+from airflow.models import BaseOperator, TaskInstance
+from airflow.hooks.base_hook
import BaseHook
+from airflow.hooks.hdfs_hook
import HDFSHook
+from airflow.utils.state import State
+from airflow.utils.decorators
import apply_defaults
class BaseSensorOperator(BaseOperator):
@@ -193,7 +221,7 @@
self,
poke_interval=60,
timeout=60*60*24*7,
-soft_fail=False,
+soft_fail=False,
*args, **kwargs):
super(BaseSensorOperator, self).__init__(*args, **kwargs)
self.poke_interval = poke_interval
@@ -245,13 +273,13 @@
logging.info('Poking:
' + self.sql)
records = hook.get_records(self.sql)
if not records:
-return False
+return False
else:
if str(records[0][0]) in ('0', '',):
-return False
+return False
else:
-return True
-print(records[0][0])
+return True
+print(records[0][0])
[docs]class MetastorePartitionSensor(SqlSensor):
@@ -286,13 +314,18 @@
self.partition_name = partition_name
self.table = table
self.schema = schema
-self.first_poke = True
+self.first_poke = True
self.conn_id = mysql_conn_id
+# TODO(aoen): We shouldn't be using SqlSensor
here but MetastorePartitionSensor.
+# The problem is the way apply_defaults works
isn't compatible with inheritance.
+# The inheritance model needs to be reworked in order
to support overriding args/
+# kwargs with arguments here, then 'conn_id'
and 'sql' can be passed into the
+# constructor below and apply_defaults will no longer
throw an exception.
super(SqlSensor, self).__init__(*args, **kwargs)
def poke(self, context):
if self.first_poke:
-self.first_poke = False
+self.first_poke = False
if '.'
in self.table:
self.schema, self.table
= self.table.split('.')
self.sql = """
@@ -301,9 +334,9 @@
LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID =
B0.TBL_ID
LEFT OUTER JOIN DBS C0 ON B0.DB_ID =
C0.DB_ID
WHERE
-B0.TBL_NAME = '{self.table}'
AND
-C0.NAME = '{self.schema}' AND
-A0.PART_NAME =
'{self.partition_name}';
+B0.TBL_NAME = '{self.table}' AND
+C0.NAME = '{self.schema}' AND
+A0.PART_NAME = '{self.partition_name}';
""".format(self=self)
return super(MetastorePartitionSensor, self).poke(context)
@@ -338,13 +371,13 @@
self,
external_dag_id,
external_task_id,
-allowed_states=None,
-execution_delta=None,
-
[15/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/dbapi_hook.html
--
diff --git a/_modules/dbapi_hook.html b/_modules/dbapi_hook.html
index 604..ae37fd3 100644
--- a/_modules/dbapi_hook.html
+++ b/_modules/dbapi_hook.html
@@ -30,6 +30,9 @@
+
+
@@ -40,6 +43,7 @@
+
@@ -90,6 +94,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -104,8 +110,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -118,19 +126,36 @@
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
+
+ Docs »
+
Module code »
-
-dbapi_hook
+
+ dbapi_hook
+
+
-
+
+
+
+
http://schema.org/Article";>
@@ -151,15 +176,17 @@
# See the License for the specific language governing
permissions and
# limitations under the License.
-from builtins import str
-from past.builtins import basestring
-from datetime import datetime
+from builtins import str
+from past.builtins import basestring
+from datetime import datetime
import numpy
import logging
import sys
-from airflow.hooks.base_hook
import BaseHook
-from airflow.exceptions import AirflowException
+from sqlalchemy import create_engine
+
+from airflow.hooks.base_hook
import BaseHook
+from airflow.exceptions import AirflowException
[docs]class DbApiHook(BaseHook):
@@ -167,13 +194,13 @@
Abstract base class for sql hooks.
"""
# Override to provide the connection name.
-conn_name_attr = None
+conn_name_attr = None
# Override to have a default connection id for a
particular dbHook
default_conn_name = 'default_conn_id'
# Override if this db supports autocommit.
-supports_autocommit = False
+supports_autocommit = False
# Override with the object that exposes the connect
method
-connector = None
+connector = None
def __init__(self, *args, **kwargs):
if not self.conn_name_attr:
@@ -195,7 +222,23 @@
username=db.login,
schema=db.schema)
-[docs]def get_pandas_df(self, sql, parameters=None):
+def get_uri(self):
+conn = self.get_connection(getattr(self, self.conn_name_attr))
+login = ''
+if conn.login:
+login = '{conn.login}:{conn.password}@'.format(conn=conn)
+host = conn.host
+if conn.port is not None:
+host += ':{port}'.format(port=conn.port)
+return '{conn.conn_type}://{login}{host}/{conn.schema}'.format(
+conn=conn, login=login, host=host)
+
+def get_sqlalchemy_engine(self, engine_kwargs=None):
+if engine_kwargs is None:
+engine_kwargs =
{}
+return create_engine(self.get_uri(), **engine_kwargs)
+
+[docs]def get_pandas_df(self, sql, parameters=None):
"""
Executes the sql and returns a pandas dataframe
@@ -207,13 +250,13 @@
"""
if sys.version_info[0] < 3:
sql = sql.encode('utf-8')
-import pandas.io.sql
as psql
+import pandas.io.sql
as psql
conn = self.get_conn()
df = psql.read_sql(sql, con=conn, params=parameters)
conn.close()
return df
-[docs]def get_records(self, sql, parameters=None):
+[docs]def get_records(self, sql, parameters=None):
"""
Executes the sql and returns a set of records.
@@ -227,7 +270,7 @@
sql = sql.encode('utf-8')
conn = self.get_conn()
cur = self.get_cursor()
-if parameters is not None:
+if parameters is not None:
cur.execute(sql, parameters)
else:
cur.execute(sql)
@@ -236,7 +279,7 @@
conn.close()
return rows
-[docs]def get_first(self, sql, parameters=None):
+[docs]def get_first(self, sql, parameters=None):
"""
Executes the sql and returns the first resulting
row.
@@ -250,7 +293,7 @@
sql = sql.encode('utf-8')
conn = self.get_conn()
cur = conn.cursor()
-if parameters is not None:
+if parameters is not None:
cur.execute(sql, parameters)
else:
cur.execute(sql)
@@ -259,7 +302,7 @@
conn.close()
return rows
-[docs]def run(self, sql, autocommit=False, parameters=None):
+[docs]def run(self, sql, autocommit=False, parameters=None):
"""
Runs a command or a list of commands. Pass a list of
sql
statements to the sql parameter to get them to
execute
@@ -275,7 +318,7 @@
:type parameters: mapping or iterable
"""
conn = self.get_conn()
-if isinstance(sql, basestring):
+if isinstance(sql, basestri
[08/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.woff
--
diff --git a/_static/fonts/fontawesome-webfont.woff
b/_static/fonts/fontawesome-webfont.woff
index 628b6a5..6e7483c 100644
Binary files a/_static/fonts/fontawesome-webfont.woff and
b/_static/fonts/fontawesome-webfont.woff differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/jquery.js
--
diff --git a/_static/jquery.js b/_static/jquery.js
index ab28a24..f6a6a99 100644
--- a/_static/jquery.js
+++ b/_static/jquery.js
@@ -1,4 +1,4 @@
-/*! jQuery v1.11.1 | (c) 2005, 2014 jQuery Foundation, Inc. |
jquery.org/license */
-!function(a,b){"object"==typeof module&&"object"==typeof
module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw
new Error("jQuery requires a window with a document");return
b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){var
c=[],d=c.slice,e=c.concat,f=c.push,g=c.indexOf,h={},i=h.toString,j=h.hasOwnProperty,k={},l="1.11.1",m=function(a,b){return
new
m.fn.init(a,b)},n=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,o=/^-ms-/,p=/-([\da-z])/gi,q=function(a,b){return
b.toUpperCase()};m.fn=m.prototype={jquery:l,constructor:m,selector:"",length:0,toArray:function(){return
d.call(this)},get:function(a){return
null!=a?0>a?this[a+this.length]:this[a]:d.call(this)},pushStack:function(a){var
b=m.merge(this.constructor(),a);return
b.prevObject=this,b.context=this.context,b},each:function(a,b){return
m.each(this,a,b)},map:function(a){return
this.pushStack(m.map(this,function(b,c){return
a.call(b,c,b)}))},slice:function(){return this.pushStack(d.apply(this,argumen
ts))},first:function(){return this.eq(0)},last:function(){return
this.eq(-1)},eq:function(a){var b=this.length,c=+a+(0>a?b:0);return
this.pushStack(c>=0&&b>c?[this[c]]:[])},end:function(){return
this.prevObject||this.constructor(null)},push:f,sort:c.sort,splice:c.splice},m.extend=m.fn.extend=function(){var
a,b,c,d,e,f,g=arguments[0]||{},h=1,i=arguments.length,j=!1;for("boolean"==typeof
g&&(j=g,g=arguments[h]||{},h++),"object"==typeof
g||m.isFunction(g)||(g={}),h===i&&(g=this,h--);i>h;h++)if(null!=(e=arguments[h]))for(d
in
e)a=g[d],c=e[d],g!==c&&(j&&c&&(m.isPlainObject(c)||(b=m.isArray(c)))?(b?(b=!1,f=a&&m.isArray(a)?a:[]):f=a&&m.isPlainObject(a)?a:{},g[d]=m.extend(j,f,c)):void
0!==c&&(g[d]=c));return
g},m.extend({expando:"jQuery"+(l+Math.random()).replace(/\D/g,""),isReady:!0,error:function(a){throw
new
Error(a)},noop:function(){},isFunction:function(a){return"function"===m.type(a)},isArray:Array.isArray||function(a){return"array"===m.type(a)},isWindow:function(a){return
null!=a&&a=
=a.window},isNumeric:function(a){return!m.isArray(a)&&a-parseFloat(a)>=0},isEmptyObject:function(a){var
b;for(b in a)return!1;return!0},isPlainObject:function(a){var
b;if(!a||"object"!==m.type(a)||a.nodeType||m.isWindow(a))return!1;try{if(a.constructor&&!j.call(a,"constructor")&&!j.call(a.constructor.prototype,"isPrototypeOf"))return!1}catch(c){return!1}if(k.ownLast)for(b
in a)return j.call(a,b);for(b in a);return void
0===b||j.call(a,b)},type:function(a){return null==a?a+"":"object"==typeof
a||"function"==typeof a?h[i.call(a)]||"object":typeof
a},globalEval:function(b){b&&m.trim(b)&&(a.execScript||function(b){a.eval.call(a,b)})(b)},camelCase:function(a){return
a.replace(o,"ms-").replace(p,q)},nodeName:function(a,b){return
a.nodeName&&a.nodeName.toLowerCase()===b.toLowerCase()},each:function(a,b,c){var
d,e=0,f=a.length,g=r(a);if(c){if(g){for(;f>e;e++)if(d=b.apply(a[e],c),d===!1)break}else
for(e in a)if(d=b.apply(a[e],c),d===!1)break}else
if(g){for(;f>e;e++)if(d=b.call(a[e],e,a[e]),d
===!1)break}else for(e in a)if(d=b.call(a[e],e,a[e]),d===!1)break;return
a},trim:function(a){return
null==a?"":(a+"").replace(n,"")},makeArray:function(a,b){var c=b||[];return
null!=a&&(r(Object(a))?m.merge(c,"string"==typeof
a?[a]:a):f.call(c,a)),c},inArray:function(a,b,c){var d;if(b){if(g)return
g.call(b,a,c);for(d=b.length,c=c?0>c?Math.max(0,d+c):c:0;d>c;c++)if(c in
b&&b[c]===a)return c}return-1},merge:function(a,b){var
c=+b.length,d=0,e=a.length;while(c>d)a[e++]=b[d++];if(c!==c)while(void
0!==b[d])a[e++]=b[d++];return a.length=e,a},grep:function(a,b,c){for(var
d,e=[],f=0,g=a.length,h=!c;g>f;f++)d=!b(a[f],f),d!==h&&e.push(a[f]);return
e},map:function(a,b,c){var
d,f=0,g=a.length,h=r(a),i=[];if(h)for(;g>f;f++)d=b(a[f],f,c),null!=d&&i.push(d);else
for(f in a)d=b(a[f],f,c),null!=d&&i.push(d);return
e.apply([],i)},guid:1,proxy:function(a,b){var c,e,f;return"string"==typeof
b&&(f=a[b],b=a,a=f),m.isFunction(a)?(c=d.call(arguments,2),e=function(){return
a.apply(b||this,c.concat(d.call(ar
guments)))},e.guid=a.guid=a.guid||m.guid++,e):void
0},now:function(){return+new Date},support:k}),m.each("Boolean Number String
F
[20/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_static/jquery-3.1.0.js
--
diff --git a/_static/jquery-3.1.0.js b/_static/jquery-3.1.0.js
new file mode 100644
index 000..f2fc274
--- /dev/null
+++ b/_static/jquery-3.1.0.js
@@ -0,0 +1,10074 @@
+/*eslint-disable no-unused-vars*/
+/*!
+ * jQuery JavaScript Library v3.1.0
+ * https://jquery.com/
+ *
+ * Includes Sizzle.js
+ * https://sizzlejs.com/
+ *
+ * Copyright jQuery Foundation and other contributors
+ * Released under the MIT license
+ * https://jquery.org/license
+ *
+ * Date: 2016-07-07T21:44Z
+ */
+( function( global, factory ) {
+
+ "use strict";
+
+ if ( typeof module === "object" && typeof module.exports === "object" )
{
+
+ // For CommonJS and CommonJS-like environments where a proper
`window`
+ // is present, execute the factory and get jQuery.
+ // For environments that do not have a `window` with a
`document`
+ // (such as Node.js), expose a factory as module.exports.
+ // This accentuates the need for the creation of a real
`window`.
+ // e.g. var jQuery = require("jquery")(window);
+ // See ticket #14549 for more info.
+ module.exports = global.document ?
+ factory( global, true ) :
+ function( w ) {
+ if ( !w.document ) {
+ throw new Error( "jQuery requires a
window with a document" );
+ }
+ return factory( w );
+ };
+ } else {
+ factory( global );
+ }
+
+// Pass this if window is not defined yet
+} )( typeof window !== "undefined" ? window : this, function( window, noGlobal
) {
+
+// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 -
9.1
+// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict
mode
+// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict
mode should be common
+// enough that all such attempts are guarded in a try block.
+"use strict";
+
+var arr = [];
+
+var document = window.document;
+
+var getProto = Object.getPrototypeOf;
+
+var slice = arr.slice;
+
+var concat = arr.concat;
+
+var push = arr.push;
+
+var indexOf = arr.indexOf;
+
+var class2type = {};
+
+var toString = class2type.toString;
+
+var hasOwn = class2type.hasOwnProperty;
+
+var fnToString = hasOwn.toString;
+
+var ObjectFunctionString = fnToString.call( Object );
+
+var support = {};
+
+
+
+ function DOMEval( code, doc ) {
+ doc = doc || document;
+
+ var script = doc.createElement( "script" );
+
+ script.text = code;
+ doc.head.appendChild( script ).parentNode.removeChild( script );
+ }
+/* global Symbol */
+// Defining this global in .eslintrc would create a danger of using the global
+// unguarded in another place, it seems safer to define global only for this
module
+
+
+
+var
+ version = "3.1.0",
+
+ // Define a local copy of jQuery
+ jQuery = function( selector, context ) {
+
+ // The jQuery object is actually just the init constructor
'enhanced'
+ // Need init if jQuery is called (just allow error to be thrown
if not included)
+ return new jQuery.fn.init( selector, context );
+ },
+
+ // Support: Android <=4.0 only
+ // Make sure we trim BOM and NBSP
+ rtrim = /^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,
+
+ // Matches dashed string for camelizing
+ rmsPrefix = /^-ms-/,
+ rdashAlpha = /-([a-z])/g,
+
+ // Used by jQuery.camelCase as callback to replace()
+ fcamelCase = function( all, letter ) {
+ return letter.toUpperCase();
+ };
+
+jQuery.fn = jQuery.prototype = {
+
+ // The current version of jQuery being used
+ jquery: version,
+
+ constructor: jQuery,
+
+ // The default length of a jQuery object is 0
+ length: 0,
+
+ toArray: function() {
+ return slice.call( this );
+ },
+
+ // Get the Nth element in the matched element set OR
+ // Get the whole matched element set as a clean array
+ get: function( num ) {
+ return num != null ?
+
+ // Return just the one element from the set
+ ( num < 0 ? this[ num + this.length ] : this[ num ] ) :
+
+ // Return all the elements in a clean array
+ slice.call( this );
+ },
+
+ // Take an array of elements and push it onto the stack
+ // (returning the new matched element set)
+ pushStack: function( elems ) {
+
+ // Build a new jQuery matched element set
+ var ret = jQuery.merge( this.const
[11/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/css/theme.css
--
diff --git a/_static/css/theme.css b/_static/css/theme.css
index 7be9339..c1631d8 100644
--- a/_static/css/theme.css
+++ b/_static/css/theme.css
@@ -1,5 +1,5 @@
*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}audio:not([controls]){display:none}[hidden]{display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:hover,a:active{outline:0}abbr[title]{border-bottom:1px
dotted}b,strong{font-weight:bold}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;color:#000;text-decoration:none}mark{background:#ff0;color:#000;font-style:italic;font-weight:bold}pre,code,.rst-content
tt,.rst-content
code,kbd,samp{font-family:monospace,serif;_font-family:"courier
new",monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:before,q:after{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;ver
tical-align:baseline}sup{top:-0.5em}sub{bottom:-0.25em}ul,ol,dl{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure{margin:0}form{margin:0}fieldset{border:0;margin:0;padding:0}label{cursor:pointer}legend{border:0;*margin-left:-7px;padding:0;white-space:normal}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type="button"],input[type="reset"],input[type="submit"]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0;*width:13px;*height:13px}input[type="search"]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}input[type="search"]::-webkit-search-decor
ation,input[type="search"]::-webkit-search-cancel-button{-webkit-appearance:none}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}textarea{overflow:auto;vertical-align:top;resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:0.2em
0;background:#ccc;color:#000;padding:0.2em
0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir
br{display:none}.hidden{display:none
!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0
0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media
print{html,body,section{background:none !important}*{box-shadow:n
one !important;text-shadow:none !important;filter:none
!important;-ms-filter:none !important}a,a:visited{text-decoration:underline}.ir
a:after,a[href^="javascript:"]:after,a[href^="#"]:after{content:""}pre,blockquote{page-break-inside:avoid}thead{display:table-header-group}tr,img{page-break-inside:avoid}img{max-width:100%
!important}@page{margin:0.5cm}p,h2,.rst-content .toctree-wrapper
p.caption,h3{orphans:3;widows:3}h2,.rst-content .toctree-wrapper
p.caption,h3{page-break-after:avoid}}.fa:before,.wy-menu-vertical li
span.toctree-expand:before,.wy-menu-vertical li.on a
span.toctree-expand:before,.wy-menu-vertical li.current>a
span.toctree-expand:before,.rst-content .admonition-title:before,.rst-content
h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3
.headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5
.headerlink:before,.rst-content h6 .headerlink:before,.rst-content dl dt
.headerlink:before,.rst-content p.caption .headerlink:before,.rst-con
tent tt.download span:first-child:before,.rst-content code.download
span:first-child:before,.icon:before,.wy-dropdown
.caret:before,.wy-inline-validate.wy-inline-validate-success
.wy-input-context:before,.wy-inline-validate.wy-inline-validate-danger
.wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning
.wy-input-context:before,.wy-inline-validate.wy-inline-validate-info
.wy-input-context:before,.wy-alert,.rst-content .note,.rst-content
.attention,.rst-content .caution,.rst-content .danger,.rst-content
.error,.rst-content .hint,.rst-content .important,.rst-content
.tip,.rst-content .warning,.rst-content .seealso,
[09/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.svg -- diff --git a/_static/fonts/fontawesome-webfont.svg b/_static/fonts/fontawesome-webfont.svg index d907b25..8b66187 100644 --- a/_static/fonts/fontawesome-webfont.svg +++ b/_static/fonts/fontawesome-webfont.svg @@ -147,14 +147,14 @@ - + - + @@ -169,7 +169,7 @@ - + @@ -178,7 +178,7 @@ - + @@ -219,8 +219,8 @@ - - + + @@ -275,7 +275,7 @@ - + @@ -362,8 +362,8 @@ - - + + @@ -399,7 +399,7 @@ - + @@ -410,9 +410,9 @@ - - - + + + @@ -438,7 +438,7 @@ - + @@ -454,12 +454,12 @@ - + - + @@ -483,13 +483,13 @@ - - + + - + @@ -513,8 +513,173 @@ - - + + + + + + + + + +http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.ttf -- diff --git a/_static/fonts/fontawesome-webfont.ttf b/_static/fonts/fontawesome-webfont.ttf index 96a3639..f221e50 100644 Binary files a/_static/fonts/fontawesome-webfont.ttf and b/_static/fonts/fontawesome-webfont.ttf differ
[12/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/ssh_hook.html
--
diff --git a/_modules/ssh_hook.html b/_modules/ssh_hook.html
index e094b62..8da4b35 100644
--- a/_modules/ssh_hook.html
+++ b/_modules/ssh_hook.html
@@ -30,6 +30,9 @@
+
+
@@ -40,6 +43,7 @@
+
@@ -90,6 +94,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -104,8 +110,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -118,19 +126,36 @@
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
+
+ Docs »
+
Module code »
-
-ssh_hook
+
+ ssh_hook
+
+
-
+
+
+
+
http://schema.org/Article";>
@@ -156,10 +181,10 @@
#
# This is a port of Luigi's ssh implementation. All
credits go there.
import subprocess
-from contextlib import contextmanager
+from contextlib import contextmanager
-from airflow.hooks.base_hook
import BaseHook
-from airflow.exceptions import AirflowException
+from airflow.hooks.base_hook
import BaseHook
+from airflow.exceptions import AirflowException
import logging
@@ -191,11 +216,13 @@
"""
def __init__(self, conn_id='ssh_default'):
conn = self.get_connection(conn_id)
-self.key_file = conn.extra_dejson.get('key_file', None)
-self.connect_timeout = conn.extra_dejson.get('connect_timeout', None)
-self.no_host_key_check = conn.extra_dejson.get('no_host_key_check', False)
-self.tty = conn.extra_dejson.get('tty', False)
-self.sshpass = conn.extra_dejson.get('sshpass', False)
+self.key_file = conn.extra_dejson.get('key_file', None)
+self.connect_timeout = conn.extra_dejson.get('connect_timeout', None)
+self.tcp_keepalive = conn.extra_dejson.get('tcp_keepalive', False)
+self.server_alive_interval = conn.extra_dejson.get('server_alive_interval', 60)
+self.no_host_key_check = conn.extra_dejson.get('no_host_key_check', False)
+self.tty = conn.extra_dejson.get('tty', False)
+self.sshpass = conn.extra_dejson.get('sshpass', False)
self.conn = conn
def get_conn(self):
@@ -203,7 +230,7 @@
def _host_ref(self):
if self.conn.login:
-return "{0}@{1}".format(self.conn.login, self.conn.host)
+return "{0}@{1}".format(self.conn.login, self.conn.host)
else:
return self.conn.host
@@ -218,7 +245,11 @@
connection_cmd +=
["-p", str(self.conn.port)]
if self.connect_timeout:
-connection_cmd +=
["-o", "ConnectionTimeout={}".format(self.connect_timeout)]
+connection_cmd +=
["-o", "ConnectionTimeout={}".format(self.connect_timeout)]
+
+if self.tcp_keepalive:
+connection_cmd +=
["-o", "TCPKeepAlive=yes"]
+connection_cmd +=
["-o", "ServerAliveInterval={}".format(self.server_alive_interval)]
if self.no_host_key_check:
connection_cmd +=
["-o", "UserKnownHostsFile=/dev/null",
@@ -231,7 +262,7 @@
connection_cmd +=
["-t"]
connection_cmd += cmd
-logging.debug("SSH cmd:
{} ".format(connection_cmd))
+logging.debug("SSH cmd:
{} ".format(connection_cmd))
return connection_cmd
@@ -259,13 +290,13 @@
if p.returncode !=
0:
# I like this better:
RemoteCalledProcessError(p.returncode, cmd, self.host, output=output)
-raise AirflowException("Cannot execute {} on {}. Error code is: {}. Output: {},
Stderr: {}".format(
+raise AirflowException("Cannot execute {} on {}. Error code
is: {}. Output: {}, Stderr: {}".format(
cmd, self.conn.host, p.returncode, output, stderr))
return output
@contextmanager
-[docs]def tunnel(self, local_port, remote_port=None, remote_host="localhost"):
+[docs]def tunnel(self, local_port, remote_port=None, remote_host="localhost"):
"""
Creates a tunnel between two hosts. Like ssh -L
:host:.
Remember to close() the returned "tunnel"
object in order to clean up
@@ -279,7 +310,7 @@
:type remote_host: str
:return:
"""
-tunnel_host = "{0}:{1}:{2}".format(local_port, remote_host, remote_port)
+tunnel_host = "{0}:{1}:{2}".format(local_port, remote_host, remote_port)
proc = self.Popen(["-L", tunnel_host,
"echo -n ready && cat"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE)
@@ -287,11 +318,14 @@
assert ready == b"ready", "Did not get 'ready' from remote"
yield
proc.communicate()
-assert proc.returncod
[03/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/plugins.html
--
diff --git a/plugins.html b/plugins.html
index 68c615d..be1cfbb 100644
--- a/plugins.html
+++ b/plugins.html
@@ -30,6 +30,9 @@
+
+
@@ -41,6 +44,7 @@
+
@@ -97,6 +101,8 @@
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -111,8 +117,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -121,23 +129,40 @@
-
+
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
-Plugins
+
+ Docs »
+
+ Plugins
+
+
-
- View page source
+
+ View page
source
+
+
+
http://schema.org/Article";>
@@ -226,18 +251,22 @@ definitions in Airflow.
from airflow.models import BaseOperator
from airflow.executors.base_executor import
BaseExecutor
-# Will show up under airflow.hooks.PluginHook
+# Will show up under
airflow.hooks.test_plugin.PluginHook
class PluginHook(BaseHook):
pass
-# Will show up under airflow.operators.PluginOperator
+# Will show up under
airflow.operators.test_plugin.PluginOperator
class PluginOperator(BaseOperator):
pass
-# Will show up under airflow.executors.PluginExecutor
+# Will show up under
airflow.executors.test_plugin.PluginExecutor
class PluginExecutor(BaseExecutor):
pass
+# Will show up under
airflow.macros.test_plugin.plugin_macro
+def plugin_macro():
+pass
+
# Creating a flask admin BaseView
class TestView(BaseView):
@expose('/')
@@ -262,10 +291,11 @@ definitions in Airflow.
class AirflowTestPlugin(AirflowPlugin):
name = "test_plugin"
operators = [PluginOperator]
-flask_blueprints = [bp]
hooks = [PluginHook]
executors = [PluginExecutor]
+macros = [plugin_macro]
admin_views = [v]
+flask_blueprints = [bp]
menu_links = [ml]
@@ -274,15 +304,18 @@ definitions in Airflow.
+
+
+
-Next
+Next
-
Previous
+ Previous
@@ -315,7 +348,8 @@ definitions in Airflow.
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
-HAS_SOURCE: true
+HAS_SOURCE: true,
+SOURCELINK_SUFFIX: '.txt'
};
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/profiling.html
--
diff --git a/profiling.html b/profiling.html
index 795ec81..350c015 100644
--- a/profiling.html
+++ b/profiling.html
@@ -30,6 +30,9 @@
+
+
@@ -41,6 +44,7 @@
+
@@ -99,6 +103,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -113,8 +119,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -123,23 +131,40 @@
-
+
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
-Data Profiling
+
+ Docs »
+
+ Data Profiling
+
+
-
- View page
source
+
+ View page
source
+
+
+
http://schema.org/Article";>
@@ -181,15 +206,18 @@ directly in the URL.
+
+
+
-Next
+Next
- Previous
+
Previous
@@ -222,7 +250,8 @@ directly in the URL.
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
-HAS_SOURCE: true
+HAS_SOURCE: true,
+SOURCELINK_SUFFIX: '.txt'
};
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/project.html
--
diff --git a/project.html b/project.html
index d2eb7f2..1bc09d0 100644
--- a/project.html
+++ b/project.html
@@ -30,6 +30,9 @@
+
+
@@ -41,6 +44,7 @@
+
@@ -97,6 +101,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -111,8 +117,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -121,23 +129,40 @@
-
+
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
-Project
+
+ Docs »
+
+ Project
+
[04/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/genindex.html -- diff --git a/genindex.html b/genindex.html index d4a80fb..76acd11 100644 --- a/genindex.html +++ b/genindex.html @@ -31,6 +31,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,17 +126,34 @@ + + + + + + + + + + + -Docs » - - + + Docs » + + + + - + + + + http://schema.org/Article";> @@ -153,1167 +178,536 @@ | N | O | P - | Q | R | S | T | U - | V | W | X A - - - add_task() (airflow.models.DAG method) - - - - add_tasks() (airflow.models.DAG method) - - - - airflow.contrib.hooks (module) - - - - airflow.contrib.operators (module) - - - - airflow.executors (module) - - - - airflow.hooks (module) - - - - - - airflow.macros (module) - - - - airflow.macros.hive (module) - - - - airflow.models (module) - - - - airflow.operators (module) - - - - are_dependencies_met() (airflow.models.TaskInstance method) - - - - are_dependents_done() (airflow.models.TaskInstance method) - - - + + add_task() (airflow.models.DAG method) + + add_tasks() (airflow.models.DAG method) + + airflow.contrib.hooks (module) + + airflow.contrib.operators (module) + + airflow.executors (module) + + airflow.hooks (module) + + + + airflow.macros (module) + + airflow.macros.hive (module) + + airflow.models (module) + + airflow.operators (module) + + are_dependencies_met() (airflow.models.TaskInstance method) + + are_dependents_done() (airflow.models.TaskInstance method) + + B - - - bag_dag() (airflow.models.DagBag method) - - - - BaseOperator (class in airflow.models), [1] - - - - BaseSensorOperator (class in airflow.operators.sensors) - - - - BashOperator (class in airflow.operators) - - - - BigQueryHook (class in airflow.contrib.hooks) - - - - - - BigQueryOperator (class in airflow.contrib.operators.bigquery_operator) - - - - BigQueryToCloudStorageOperator (class in airflow.contrib.operators.bigquery_to_gcs) - - - - BranchPythonOperator (class in airflow.operators) - - - - bulk_dump() (airflow.hooks.DbApiHook method) - - - - bulk_load() (airflow.hooks.DbApiHook method) - - - - - (airflow.hooks.MySqlHook method) - - - - + + bag_dag() (airflow.models.DagBag method) + + BaseOperator (class in airflow.models), [1] + + BaseSensorOperator (class in airflow.operators.sensors) + + BashOperator (class in airflow.operators) + + + + BranchPythonOperator (class in airflow.operators) + + bulk_dump() (airflow.hooks.DbApiHook method) + + bulk_load() (airflow.hooks.DbApiHook method) + + +(airflow.hooks.MySqlHook method) + + + C - - - CeleryExecutor (class in airflow.executors) - - - - check_for_bucket() (airflow.hooks.S3Hook method) - - - - check_for_key() (airflow.hooks.S3Hook method) - - - - check_for_named_partition() (airflow.hooks.HiveMetastoreHook method) - - - - check_for_partition() (airflow.hooks.HiveMetastoreHook method) - - - - check_for_path() (airflow.hooks.WebHDFSHook method) - - - - check_for_prefix() (airflow.hooks.S3Hook method) - - - - check_for_wildcard_key() (airflow.hooks.S3Hook method) - - - - check_output() (airflow.contrib.hooks.SSHHook method) - - - - clear() (airflow.models.BaseOperator method) - - - - - (airflow.models.DAG method) - - - - - clear_xcom_data() (airflow.models.TaskInstance method) - - - - cli() (airflow.models.DAG method) - - - - close_conn() (airflow.contrib.hooks.FTPHook method) - - - - - - closest_ds_partition() (in module airflow.macros.hive) - - - - CloudantHook (class in airflow.contrib.hooks) - - - - collect_dags() (airflow.models.DagBag method) - - - - command() (airflow.models.TaskInstance method) - - - - concurrency_reached (airflow.models.DAG attribute) - - - - Connection (class in airflow.models) - - - - construct_api_call_params() (airflow.operators.SlackAPIOperator method) - - - - construct_ingest_query() (airflow.hooks.DruidHook method) - - - -
[07/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/js/theme.js
--
diff --git a/_static/js/theme.js b/_static/js/theme.js
index 48a9f06..af661a9 100644
--- a/_static/js/theme.js
+++ b/_static/js/theme.js
@@ -13,33 +13,36 @@ function ThemeNav () {
winPosition: 0,
winHeight: null,
docHeight: null,
-isRunning: null
+isRunning: false
};
nav.enable = function () {
var self = this;
-jQuery(function ($) {
-self.init($);
-
-self.reset();
-self.win.on('hashchange', self.reset);
-
-// Set scroll monitor
-self.win.on('scroll', function () {
-if (!self.linkScroll) {
-self.winScroll = true;
-}
-});
-setInterval(function () { if (self.winScroll) self.onScroll(); },
25);
-
-// Set resize monitor
-self.win.on('resize', function () {
-self.winResize = true;
+if (!self.isRunning) {
+self.isRunning = true;
+jQuery(function ($) {
+self.init($);
+
+self.reset();
+self.win.on('hashchange', self.reset);
+
+// Set scroll monitor
+self.win.on('scroll', function () {
+if (!self.linkScroll) {
+self.winScroll = true;
+}
+});
+setInterval(function () { if (self.winScroll) self.onScroll();
}, 25);
+
+// Set resize monitor
+self.win.on('resize', function () {
+self.winResize = true;
+});
+setInterval(function () { if (self.winResize) self.onResize();
}, 25);
+self.onResize();
});
-setInterval(function () { if (self.winResize) self.onResize(); },
25);
-self.onResize();
-});
+};
};
nav.init = function ($) {
@@ -95,6 +98,19 @@ function ThemeNav () {
try {
var link = $('.wy-menu-vertical')
.find('[href="' + anchor + '"]');
+// If we didn't find a link, it may be because we clicked on
+// something that is not in the sidebar (eg: when using
+// sphinxcontrib.httpdomain it generates headerlinks but those
+// aren't picked up and placed in the toctree). So let's find
+// the closest header in the document and try with that one.
+if (link.length === 0) {
+ var doc_link = $('.document a[href="' + anchor + '"]');
+ var closest_section = doc_link.closest('div.section');
+ // Try again with the closest section entry.
+ link = $('.wy-menu-vertical')
+.find('[href="#' + closest_section.attr("id") + '"]');
+
+}
$('.wy-menu-vertical li.toctree-l1 li.current')
.removeClass('current');
link.closest('li.toctree-l2').addClass('current');
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/minus.png
--
diff --git a/_static/minus.png b/_static/minus.png
index 0f22b16..d96755f 100644
Binary files a/_static/minus.png and b/_static/minus.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/plus.png
--
diff --git a/_static/plus.png b/_static/plus.png
index 0cfe084..7107cec 100644
Binary files a/_static/plus.png and b/_static/plus.png differ
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/searchtools.js
--
diff --git a/_static/searchtools.js b/_static/searchtools.js
index 066857c..bbfb3ac 100644
--- a/_static/searchtools.js
+++ b/_static/searchtools.js
@@ -226,6 +226,106 @@ var Scorer = {
};
+
+
+
+var splitChars = (function() {
+var result = {};
+var singles = [96, 180, 187, 191, 215, 247, 749, 885, 903, 907, 909, 930,
1014, 1648,
+ 1748, 1809, 2416, 2473, 2481, 2526, 2601, 2609, 2612, 2615, 2653,
2702,
+ 2706, 2729, 2737, 2740, 2857, 2865, 2868, 2910, 2928, 2948, 2961,
2971,
+ 2973, 3085, 3089, 3113, 3124, 3213, 3217, 3241, 3252, 3295, 3341,
3345,
+ 3369, 3506, 3516, 3633, 3715, 3721, 3736, 3744, 3748, 3750, 3756,
3761,
+ 3781, 3912, 4239, 4347, 4681, 4695, 4697, 4745, 4785, 4799, 4801,
4823,
+ 4881, 5760, 5901, 5997, 6313, 7405, 8024, 8026, 8028, 8030, 8117,
8125,
+ 8133, 8181, 8468, 8485, 8487, 8489, 8494, 8527, 11311, 11359, 11687,
11695,
+ 11703, 117
[06/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/code.html -- diff --git a/code.html b/code.html index 91192d3..732db2e 100644 --- a/code.html +++ b/code.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,13 +94,15 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference Operators BaseOperator BaseSensorOperator -Operator API -Community-contributed Operators +Operator API +Community-contributed Operators Macros @@ -105,8 +111,8 @@ Models -Hooks -Community contributed hooks +Hooks +Community contributed hooks Executors @@ -127,8 +133,10 @@ - -Airflow + + + Airflow + @@ -137,23 +145,40 @@ - + + + + + + + + + + + + -Docs » - -API Reference + + Docs » + + API Reference + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -187,12 +212,12 @@ to understand the primitive features that can be leveraged in your DAGs. -class airflow.models.BaseOperator(task_id, owner='airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, *args, **kwargs< /em>)[source]¶ +class airflow.models.BaseOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, *args, **kwargs)[source]¶ Abstract base class for all operators. Since operators create objects that become node in the dag, BaseOperator contains many recursive methods for dag crawling behavior. To derive this class, you are expected to override the constructor as well as the ‘execute’ method. -Operators derived from this task should perform or trigger certain tasks +Operators derived from this class should perform or trigger certain tasks synchronously (wait for completion). Example of operators could be an operator the runs a Pig job (PigOperator), a sensor operator that waits for a partition to land in Hive (HiveSensorOperator), or one that @@ -261,7 +286,7 @@ way to limit concurrency for certain tasks this represents the timedelta after the period is closed. For example if you set an SLA of 1 hour, the scheduler would send dan email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance -has not succeede yet. +has not succeeded yet. The scheduler pays special attention for jobs with an SLA and sends alert emails for sla misses. SLA misses are also recorded in the database @@ -288,6 +313,7 @@ using the constants defined in the static class airflow.utils.TriggerRule resources (dict) – A map of resource parameter names (the argument names of the Resources constructor) to their values. +run_as_user (str) – unix username to impersonate while running the task @@ -326,9 +352,9 @@ between each tries - -Operator API¶ -Importer that dynamically loads a class and module from its parent. This + +Operator API¶ +Importer that dynamically loads a class and module from its parent. This allows Airflow to support from airflow.operators import BashOperator even though BashOperator is actually in airflow.operators.bash_operator. @@ -427,7 +453,7 @@ DAG. -class airflow.operators.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, *args, **kwargs)[source]¶ +class airflow.operators.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, mime_subtype='mixed', *args, **kwargs)[source]¶ Bases: airflow.models.BaseOperator Sends an email. @@ -435,13 +461,13 @@ DAG. Parameters: -to (list or string (comma or semicolon delimited)) – list of emails to send the email to +to (list or string (comma or semicolon delimited)) – list of emails to send the email to su
[14/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/http_operator.html
--
diff --git a/_modules/http_operator.html b/_modules/http_operator.html
index 00496c9..a22b710 100644
--- a/_modules/http_operator.html
+++ b/_modules/http_operator.html
@@ -30,6 +30,9 @@
+
+
@@ -40,6 +43,7 @@
+
@@ -90,6 +94,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -104,8 +110,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -118,19 +126,36 @@
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
+
+ Docs »
+
Module code »
-
-http_operator
+
+ http_operator
+
+
-
+
+
+
+
http://schema.org/Article";>
@@ -153,10 +178,10 @@
import logging
-from airflow.exceptions import AirflowException
-from airflow.hooks.http_hook
import HttpHook
-from airflow.models import BaseOperator
-from airflow.utils.decorators
import apply_defaults
+from airflow.exceptions import AirflowException
+from airflow.hooks.http_hook
import HttpHook
+from airflow.models import BaseOperator
+from airflow.utils.decorators
import apply_defaults
[docs]class SimpleHttpOperator(BaseOperator):
@@ -192,11 +217,16 @@
def __init__(self,
endpoint,
method='POST',
- data=None,
- headers=None,
- response_check=None,
- extra_options=None,
+ data=None,
+ headers=None,
+ response_check=None,
+ extra_options=None,
+ xcom_push=False,
http_conn_id='http_default', *args, **kwargs):
+"""
+If xcom_push is True, response of an HTTP request
will also
+be pushed to an XCom.
+"""
super(SimpleHttpOperator, self).__init__(*args, **kwargs)
self.http_conn_id = http_conn_id
self.method = method
@@ -205,6 +235,7 @@
self.data = data
or {}
self.response_check = response_check
self.extra_options = extra_options or {}
+self.xcom_push_flag = xcom_push
def execute(self, context):
http = HttpHook(self.method, http_conn_id=self.http_conn_id)
@@ -215,10 +246,15 @@
self.extra_options)
if self.response_check:
if not self.response_check(response):
-raise AirflowException("Response check returned False.")
+raise AirflowException("Response check returned False.")
+if self.xcom_push_flag:
+return response.text
+
+
+
@@ -251,7 +287,8 @@
VERSION:'',
COLLAPSE_INDEX:false,
FILE_SUFFIX:'.html',
-HAS_SOURCE: true
+HAS_SOURCE: true,
+SOURCELINK_SUFFIX: '.txt'
};
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/index.html
--
diff --git a/_modules/index.html b/_modules/index.html
index d40ab64..b1faeb5 100644
--- a/_modules/index.html
+++ b/_modules/index.html
@@ -30,6 +30,9 @@
+
+
@@ -39,6 +42,7 @@
+
@@ -89,6 +93,8 @@
Scheduling & Triggers
Plugins
Security
+Experimental Rest API
+Integration
FAQ
API
Reference
@@ -103,8 +109,10 @@
-
-Airflow
+
+
+ Airflow
+
@@ -117,81 +125,72 @@
+
+
+
+
+
+
+
+
+
+
+
-Docs »
-
-Overview: module code
+
+ Docs »
+
+ Overview: module code
+
+
-
+
+
+
+
http://schema.org/Article";>
All modules for which code is available
-S3_hook
-airflow.contrib.executors.mesos_executor
-airflow.contrib.hooks.gcs_hook
-airflow.contrib.operators.bigquery_operator
-airflow.contrib.operators.bigquery_to_gcs
-airflow.contrib.operators.gcs_download_operator
+airflow.contrib.executors.mesos_executor
airflow.contrib.operators.hipchat_operator
-airflow.executors.celery_executor
airflow.executors.local_executor
airflow.executors.sequential_executor
airflow.macros
airflow.macros.hive
airflow.models
-airflow.operators.docker_operator
airflow.operators.sensors
bash_operator
-bigquery_hook
-cloudant_hook
dagrun_operator
dbapi_hook
-druid_hook
dummy_operator
email_operator
ftp_hook
-gcs_hook
generic_transfer
-hive_hooks
-hive_operator
-hive_to_druid
-hive_to
