[jira] [Created] (AIRFLOW-948) cannot run "airflow initdb"
Tony Kucera created AIRFLOW-948: --- Summary: cannot run "airflow initdb" Key: AIRFLOW-948 URL: https://issues.apache.org/jira/browse/AIRFLOW-948 Project: Apache Airflow Issue Type: Bug Affects Versions: Airflow 1.7.1 Environment: Debian Jessie 8.2, Python 2.7.9, Python 3.4.2 Reporter: Tony Kucera [2017-03-07 08:45:38,241] {__init__.py:36} INFO - Using executor SequentialExecutor [2017-03-07 08:45:38,538] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.4/lib2to3/Grammar.txt [2017-03-07 08:45:38,578] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python3.4/lib2to3/PatternGrammar.txt DB: sqlite:root/airflow/airflow.db [2017-03-07 08:45:38,898] {db.py:222} INFO - Creating tables INFO [alembic.runtime.migration] Context impl SQLiteImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. ERROR [airflow.models.DagBag] Failed to import: /usr/local/lib/python3.4/dist-packages/airflow/example_dags/example_twitter_dag.py Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/airflow/models.py", line 247, in process_file m = imp.load_source(mod_name, filepath) File "/usr/lib/python3.4/imp.py", line 171, in load_source module = methods.load() File "", line 1220, in load File "", line 1200, in _load_unlocked File "", line 1129, in _exec File "", line 1471, in exec_module File "", line 321, in _call_with_frames_removed File "/usr/local/lib/python3.4/dist-packages/airflow/example_dags/example_twitter_dag.py", line 26, in from airflow.operators import BashOperator, HiveOperator, PythonOperator ImportError: cannot import name 'HiveOperator' -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-947) Make PrestoHook surface better messages when the Presto Cluster is unavailable.
Arthur Wiedmer created AIRFLOW-947: -- Summary: Make PrestoHook surface better messages when the Presto Cluster is unavailable. Key: AIRFLOW-947 URL: https://issues.apache.org/jira/browse/AIRFLOW-947 Project: Apache Airflow Issue Type: Bug Reporter: Arthur Wiedmer Assignee: Arthur Wiedmer Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-946) Virtualenv not explicitly used by webserver/worker subprocess
Daniel Huang created AIRFLOW-946: Summary: Virtualenv not explicitly used by webserver/worker subprocess Key: AIRFLOW-946 URL: https://issues.apache.org/jira/browse/AIRFLOW-946 Project: Apache Airflow Issue Type: Bug Components: cli Reporter: Daniel Huang Assignee: Daniel Huang Priority: Minor I have airflow installed in a virtualenv. I'd expect calling {{/path/to/venv/bin/airflow webserver}} or {{/path/to/venv/bin/airflow worker}} *without activating my virtualenv* to work. However, they both fail to run properly because they spawn a process that is called without specifying the virtualenv. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-288) Make system timezone configurable
[ https://issues.apache.org/jira/browse/AIRFLOW-288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898590#comment-15898590 ] David Klosowski commented on AIRFLOW-288: - Thoughts on support for this? It would be nice to see this at the DAG level so individual DAGs could be triggered in a timezone aware way by the the scheduler. > Make system timezone configurable > - > > Key: AIRFLOW-288 > URL: https://issues.apache.org/jira/browse/AIRFLOW-288 > Project: Apache Airflow > Issue Type: Improvement >Reporter: Vineet Goel >Assignee: Vineet Goel > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-74) SubdagOperators can consume all celeryd worker processes
[ https://issues.apache.org/jira/browse/AIRFLOW-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898079#comment-15898079 ] Russell Pierce commented on AIRFLOW-74: --- This appears to still be true under Airflow 1.8.0rc4. > SubdagOperators can consume all celeryd worker processes > > > Key: AIRFLOW-74 > URL: https://issues.apache.org/jira/browse/AIRFLOW-74 > Project: Apache Airflow > Issue Type: Bug > Components: celery >Affects Versions: Airflow 1.7.1, Airflow 1.7.0, Airflow 1.6.2 > Environment: Airflow 1.7.1rc3 with CeleryExecutor > 1 webserver > 1 scheduler > 2 workers >Reporter: Steven Yvinec-Kruyk > > If the amount of concurrent ```SubdagOperator``` running >= the no. of celery > worker processes tasks are unable to work. All SDOs come to a complete halt. > Futhermore performance of a DAG is drastically reduced even before full > saturation of the workers as less workers are gradually available for actual > tasks. A workaround for this is to specify ```SequentialExecutor``` be used > by the ```SubdagOperator``` > ``` > from datetime import timedelta, datetime > from airflow.models import DAG, Pool > from airflow.operators import BashOperator, SubDagOperator, DummyOperator > from airflow.executors import SequentialExecutor > import airflow > # -\ > # DEFINE THE POOLS > # -/ > session = airflow.settings.Session() > for p in ['test_pool_1', 'test_pool_2', 'test_pool_3']: > pool = ( > session.query(Pool) > .filter(Pool.pool == p) > .first()) > if not pool: > session.add(Pool(pool=p, slots=8)) > session.commit() > # -\ > # DEFINE THE DAG > # -/ > # Define the Dag Name. This must be unique. > dag_name = 'hanging_subdags_n16_sqe' > # Default args are passed to each task > default_args = { > 'owner': 'Airflow', > 'depends_on_past': False, > 'start_date': datetime(2016, 04, 10), > 'retries': 0, > 'retry_interval': timedelta(minutes=5), > 'email': ['y...@email.com'], > 'email_on_failure': True, > 'email_on_retry': True, > 'wait_for_downstream': False, > } > # Create the dag object > dag = DAG(dag_name, > default_args=default_args, > schedule_interval='0 0 * * *' > ) > # -\ > # DEFINE THE TASKS > # -/ > def get_subdag(dag, sd_id, pool=None): > subdag = DAG( > dag_id='{parent_dag}.{sd_id}'.format( > parent_dag=dag.dag_id, > sd_id=sd_id), > params=dag.params, > default_args=dag.default_args, > template_searchpath=dag.template_searchpath, > user_defined_macros=dag.user_defined_macros, > ) > t1 = BashOperator( > task_id='{sd_id}_step_1'.format( > sd_id=sd_id > ), > bash_command='echo "hello" && sleep 60', > dag=subdag, > pool=pool, > executor=SequentialExecutor > ) > t2 = BashOperator( > task_id='{sd_id}_step_two'.format( > sd_id=sd_id > ), > bash_command='echo "hello" && sleep 15', > dag=subdag, > pool=pool, > executor=SequentialExecutor > ) > t2.set_upstream(t1) > sdo = SubDagOperator( > task_id=sd_id, > subdag=subdag, > retries=0, > retry_delay=timedelta(seconds=5), > dag=dag, > depends_on_past=True, > ) > return sdo > start_task = DummyOperator( > task_id='start', > dag=dag > ) > for n in range(1, 17): > sd_i = get_subdag(dag=dag, sd_id='level_1_{n}'.format(n=n), > pool='test_pool_1') > sd_ii = get_subdag(dag=dag, sd_id='level_2_{n}'.format(n=n), > pool='test_pool_2') > sd_iii = get_subdag(dag=dag, sd_id='level_3_{n}'.format(n=n), > pool='test_pool_3') > sd_i.set_upstream(start_task) > sd_ii.set_upstream(sd_i) > sd_iii.set_upstream(sd_ii) > ``` -- This message was sent by Atlassian JIRA (v6.3.15#6346)
incubator-airflow git commit: [AIRFLOW-941] Use defined parameters for psycopg2
Repository: incubator-airflow Updated Branches: refs/heads/master 2cfe28244 -> e79dee871 [AIRFLOW-941] Use defined parameters for psycopg2 This works around https://github.com/psycopg/psycopg2/issues/517 . Closes #2126 from bolkedebruin/AIRFLOW-941 Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow/commit/e79dee87 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow/tree/e79dee87 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow/diff/e79dee87 Branch: refs/heads/master Commit: e79dee8718ca7d69a6905509f45fd2c5e9267209 Parents: 2cfe282 Author: Bolke de Bruin Authored: Mon Mar 6 21:03:14 2017 +0100 Committer: Bolke de Bruin Committed: Mon Mar 6 21:03:14 2017 +0100 -- airflow/hooks/postgres_hook.py | 15 +++ 1 file changed, 11 insertions(+), 4 deletions(-) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow/blob/e79dee87/airflow/hooks/postgres_hook.py -- diff --git a/airflow/hooks/postgres_hook.py b/airflow/hooks/postgres_hook.py index 372e4e5..4b460c1 100644 --- a/airflow/hooks/postgres_hook.py +++ b/airflow/hooks/postgres_hook.py @@ -36,10 +36,17 @@ class PostgresHook(DbApiHook): conn = self.get_connection(self.postgres_conn_id) conn_args = dict( host=conn.host, -user=conn.login, -password=conn.password, -dbname=self.schema or conn.schema, -port=conn.port) +dbname=self.schema or conn.schema) +# work around for https://github.com/psycopg/psycopg2/issues/517 +# todo: remove when psycopg2 2.7.1 is released +# https://issues.apache.org/jira/browse/AIRFLOW-945 +if conn.port: +conn_args['port'] = conn.port +if conn.login: +conn_args['user'] = conn.login +if conn.password: +conn_args['password'] = conn.password + # check for ssl parameters in conn.extra for arg_name, arg_val in conn.extra_dejson.items(): if arg_name in ['sslmode', 'sslcert', 'sslkey', 'sslrootcert', 'sslcrl', 'application_name']:
[jira] [Commented] (AIRFLOW-941) Psycopg2 2.7.0 has a regression when port is 'None'
[ https://issues.apache.org/jira/browse/AIRFLOW-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897952#comment-15897952 ] ASF subversion and git services commented on AIRFLOW-941: - Commit e79dee8718ca7d69a6905509f45fd2c5e9267209 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e79dee8 ] [AIRFLOW-941] Use defined parameters for psycopg2 This works around https://github.com/psycopg/psycopg2/issues/517 . Closes #2126 from bolkedebruin/AIRFLOW-941 > Psycopg2 2.7.0 has a regression when port is 'None' > --- > > Key: AIRFLOW-941 > URL: https://issues.apache.org/jira/browse/AIRFLOW-941 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.8.0rc4 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin > Fix For: 1.8.0rc5 > > > workaround is to not specify username/password if they are not available. See > external issue URL. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (AIRFLOW-941) Psycopg2 2.7.0 has a regression when port is 'None'
[ https://issues.apache.org/jira/browse/AIRFLOW-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897954#comment-15897954 ] ASF subversion and git services commented on AIRFLOW-941: - Commit e79dee8718ca7d69a6905509f45fd2c5e9267209 in incubator-airflow's branch refs/heads/master from [~bolke] [ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=e79dee8 ] [AIRFLOW-941] Use defined parameters for psycopg2 This works around https://github.com/psycopg/psycopg2/issues/517 . Closes #2126 from bolkedebruin/AIRFLOW-941 > Psycopg2 2.7.0 has a regression when port is 'None' > --- > > Key: AIRFLOW-941 > URL: https://issues.apache.org/jira/browse/AIRFLOW-941 > Project: Apache Airflow > Issue Type: Bug > Components: operators >Affects Versions: 1.8.0rc4 >Reporter: Bolke de Bruin >Assignee: Bolke de Bruin > Fix For: 1.8.0rc5 > > > workaround is to not specify username/password if they are not available. See > external issue URL. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (AIRFLOW-945) Revert psycopg2 workaround when psycopg2 2.7.1 is released
Bolke de Bruin created AIRFLOW-945: -- Summary: Revert psycopg2 workaround when psycopg2 2.7.1 is released Key: AIRFLOW-945 URL: https://issues.apache.org/jira/browse/AIRFLOW-945 Project: Apache Airflow Issue Type: Bug Components: hooks Affects Versions: 1.8.0rc5 Reporter: Bolke de Bruin Fix For: 1.8.1 A workaround around for https://github.com/psycopg/psycopg2/issues/517 is present. We can revert this change when 2.7.1 is out. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[10/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/down-pressed.png -- diff --git a/_static/down-pressed.png b/_static/down-pressed.png index 7c30d00..5756c8c 100644 Binary files a/_static/down-pressed.png and b/_static/down-pressed.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/down.png -- diff --git a/_static/down.png b/_static/down.png index f48098a..1b3bdad 100644 Binary files a/_static/down.png and b/_static/down.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/file.png -- diff --git a/_static/file.png b/_static/file.png index 254c60b..a858a41 100644 Binary files a/_static/file.png and b/_static/file.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Inconsolata-Bold.ttf -- diff --git a/_static/fonts/Inconsolata-Bold.ttf b/_static/fonts/Inconsolata-Bold.ttf index 58c9fef..809c1f5 100644 Binary files a/_static/fonts/Inconsolata-Bold.ttf and b/_static/fonts/Inconsolata-Bold.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Inconsolata-Regular.ttf -- diff --git a/_static/fonts/Inconsolata-Regular.ttf b/_static/fonts/Inconsolata-Regular.ttf index a87ffba..fc981ce 100644 Binary files a/_static/fonts/Inconsolata-Regular.ttf and b/_static/fonts/Inconsolata-Regular.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Lato-Bold.ttf -- diff --git a/_static/fonts/Lato-Bold.ttf b/_static/fonts/Lato-Bold.ttf index 7434369..1d23c70 100644 Binary files a/_static/fonts/Lato-Bold.ttf and b/_static/fonts/Lato-Bold.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/Lato-Regular.ttf -- diff --git a/_static/fonts/Lato-Regular.ttf b/_static/fonts/Lato-Regular.ttf index 04ea8ef..0f3d0f8 100644 Binary files a/_static/fonts/Lato-Regular.ttf and b/_static/fonts/Lato-Regular.ttf differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.eot -- diff --git a/_static/fonts/fontawesome-webfont.eot b/_static/fonts/fontawesome-webfont.eot index 84677bc..c7b00d2 100644 Binary files a/_static/fonts/fontawesome-webfont.eot and b/_static/fonts/fontawesome-webfont.eot differ
[19/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/api.html -- diff --git a/api.html b/api.html new file mode 100644 index 000..7aca3bb --- /dev/null +++ b/api.html @@ -0,0 +1,279 @@ + + + + + + + + + + + Experimental Rest API — Airflow Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Airflow + + + + + + + + + + + + + + + + + + + + + + + + + + + +Project +License +Quick Start +Installation +Tutorial +Configuration +UI / Screenshots +Concepts +Data Profiling +Command Line Interface +Scheduling & Triggers +Plugins +Security +Experimental Rest API +Endpoints +CLI +Authentication + + +Integration +FAQ +API Reference + + + + + + + + + + + + + + + Airflow + + + + + + + + + + + + + + + + + + + + + + + + + + + + Docs » + + Experimental Rest API + + + + + + View page source + + + + + + + + + + http://schema.org/Article";> + + + +Experimental Rest API¶ +Airflow exposes an experimental Rest API. It is available through the webserver. Endpoints are +available at /api/experimental/. Please note that we expect the endpoint definitions to change. + +Endpoints¶ +This is a place holder until the swagger definitions are active + +/api/experimental/dags//tasks/ returns info for a task (GET). +/api/experimental/dags/ /dag_runs creates a dag_run for a given dag id (POST). + + + +CLI¶ +For some functions the cli can use the API. To configure the CLI to use the API when available +configure as follows: +[cli] +api_client = airflow.api.client.json_client +endpoint_url = http:// : ; + + + + +Authentication¶ +Only Kerberos authentication is currently supported for the API. To enable this set the following +in the configuration: +[api] +auth_backend = airflow.api.auth.backend.default + +[kerberos] +keytab = + + +The Kerberos service is configured as airflow/fully.qualified.domainname@REALM. Make sure this +principal exists in the keytab file. + + + + + + + + + + + + + +Next + + + Previous + + + + + + + + + + + + Built with http://sphinx-doc.org/";>Sphinx using a https://github.com/snide/sphinx_rtd_theme";>theme provided by https://readthedocs.org";>Read the Docs. + + + + + + + + + + + + + + + +var DOCUMENTATION_OPTIONS = { +URL_ROOT:'./', +VERSION:'', +COLLAPSE_INDEX:false, +FILE_SUFFIX:'.html', +HAS_SOURCE: true, +SOURCELINK_SUFFIX: '.txt' +}; + + + + + + + + + + + + + + + + jQuery(function () { + SphinxRtdTheme.StickyNav.enable(); + }); + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/integration.html -- diff --git a/integration.html b/integration.html new file mode 100644 index 000..a55aa41 --- /dev/null +++ b/integration.html @@ -0,0 +1,424 @@ + + + + + + + + + + + Integration — Airflow Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Airflow + + + + + + + + + + + + + + + + + + + + + + + + + + + +Project +License +Quick Start +Installation +Tutorial +Configuration +UI / Screenshots +Concepts +Data Profiling +Command Line Interface +Scheduling & Triggers +Plugins +Security +Experimental Rest API +Integration +AWS: Amazon Webservices +GCP: Google Cloud Platform +BigQuery +BigQuery Operators +BigQueryHook + + +Cloud DataFlow +DataFlow Operators +DataFlowHook + + +Cloud DataProc +DataProc Operators +DataProcPySparkOperator + + +Cloud Datastore +Datastore Operators + + +Cloud Storage +Storage Operators +GoogleCloudStorageHook + + + + + + +FAQ +API Reference + + + + + + + + + + + +
[18/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
Latest docs version as of 1.8.x Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/9c75ee9e Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/9c75ee9e Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/9c75ee9e Branch: refs/heads/asf-site Commit: 9c75ee9e46cc5ec459979dbfd2c911743c07fd63 Parents: 4af0850 Author: Maxime Beauchemin Authored: Mon Mar 6 08:43:12 2017 -0800 Committer: Maxime Beauchemin Committed: Mon Mar 6 08:43:12 2017 -0800 -- .../contrib/executors/mesos_executor.html | 81 +- .../contrib/operators/hipchat_operator.html | 72 +- _modules/airflow/executors/local_executor.html | 73 +- .../airflow/executors/sequential_executor.html | 59 +- _modules/airflow/macros.html| 79 +- _modules/airflow/macros/hive.html | 61 +- _modules/airflow/models.html| 1993 ++ _modules/airflow/operators/sensors.html | 242 ++- _modules/bash_operator.html | 77 +- _modules/dagrun_operator.html | 67 +- _modules/dbapi_hook.html| 130 +- _modules/dummy_operator.html| 49 +- _modules/email_operator.html| 61 +- _modules/ftp_hook.html | 69 +- _modules/generic_transfer.html | 57 +- _modules/http_hook.html | 61 +- _modules/http_operator.html | 71 +- _modules/index.html | 74 +- _modules/mysql_hook.html| 95 +- _modules/mysql_operator.html| 61 +- _modules/presto_check_operator.html | 60 +- _modules/presto_hook.html | 63 +- _modules/python_operator.html | 69 +- _modules/sensors.html | 242 ++- _modules/sqlite_hook.html | 49 +- _modules/ssh_execute_operator.html | 75 +- _modules/ssh_hook.html | 81 +- _static/basic.css | 68 +- _static/comment-bright.png | Bin 3500 -> 756 bytes _static/comment-close.png | Bin 3578 -> 829 bytes _static/comment.png | Bin 3445 -> 641 bytes _static/css/badge_only.css |2 +- _static/css/theme.css |4 +- _static/down-pressed.png| Bin 347 -> 222 bytes _static/down.png| Bin 347 -> 202 bytes _static/file.png| Bin 358 -> 286 bytes _static/fonts/Inconsolata-Bold.ttf | Bin 66352 -> 109948 bytes _static/fonts/Inconsolata-Regular.ttf | Bin 84548 -> 96964 bytes _static/fonts/Lato-Bold.ttf | Bin 121788 -> 656544 bytes _static/fonts/Lato-Regular.ttf | Bin 120196 -> 656568 bytes _static/fonts/fontawesome-webfont.eot | Bin 56006 -> 76518 bytes _static/fonts/fontawesome-webfont.svg | 207 +- _static/fonts/fontawesome-webfont.ttf | Bin 112160 -> 152796 bytes _static/fonts/fontawesome-webfont.woff | Bin 65452 -> 90412 bytes _static/jquery.js |8 +- _static/js/theme.js | 58 +- _static/minus.png | Bin 173 -> 90 bytes _static/plus.png| Bin 173 -> 90 bytes _static/searchtools.js | 115 +- _static/up-pressed.png | Bin 345 -> 214 bytes _static/up.png | Bin 345 -> 203 bytes cli.html| 482 +++-- code.html | 1933 +++-- concepts.html | 129 +- configuration.html | 97 +- faq.html| 61 +- genindex.html | 1541 -- index.html | 115 +- installation.html | 51 +- license.html| 51 +- objects.inv | Bin 2326 -> 2147 bytes plugins.html| 64 +- profiling.html | 51 +- project.html| 76 +- py-modindex.html| 63 +- scheduler.html | 79 +- search.html | 41 +- searchindex.js
[16/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/airflow/operators/sensors.html -- diff --git a/_modules/airflow/operators/sensors.html b/_modules/airflow/operators/sensors.html index 3b716d3..6d51369 100644 --- a/_modules/airflow/operators/sensors.html +++ b/_modules/airflow/operators/sensors.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,19 +126,36 @@ + + + + + + + + + + + -Docs » - + + Docs » + Module code » - -airflow.operators.sensors + + airflow.operators.sensors + + - + + + + http://schema.org/Article";> @@ -151,24 +176,27 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import print_function -from future import standard_library +from __future__ import print_function +from future import standard_library standard_library.install_aliases() -from builtins import str -from past.builtins import basestring +from builtins import str +from past.builtins import basestring -from datetime import datetime +from datetime import datetime import logging -from urllib.parse import urlparse -from time import sleep +from urllib.parse import urlparse +from time import sleep +import re +import sys import airflow -from airflow import hooks, settings -from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException -from airflow.models import BaseOperator, TaskInstance, Connection as DB -from airflow.hooks.base_hook import BaseHook -from airflow.utils.state import State -from airflow.utils.decorators import apply_defaults +from airflow import hooks, settings +from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException +from airflow.models import BaseOperator, TaskInstance +from airflow.hooks.base_hook import BaseHook +from airflow.hooks.hdfs_hook import HDFSHook +from airflow.utils.state import State +from airflow.utils.decorators import apply_defaults [docs]class BaseSensorOperator(BaseOperator): @@ -193,7 +221,7 @@ self, poke_interval=60, timeout=60*60*24*7, -soft_fail=False, +soft_fail=False, *args, **kwargs): super(BaseSensorOperator, self).__init__(*args, **kwargs) self.poke_interval = poke_interval @@ -245,13 +273,13 @@ logging.info('Poking: ' + self.sql) records = hook.get_records(self.sql) if not records: -return False +return False else: if str(records[0][0]) in ('0', '',): -return False +return False else: -return True -print(records[0][0]) +return True +print(records[0][0]) class MetastorePartitionSensor(SqlSensor): @@ -286,13 +314,18 @@ self.partition_name = partition_name self.table = table self.schema = schema -self.first_poke = True +self.first_poke = True self.conn_id = mysql_conn_id +# TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor. +# The problem is the way apply_defaults works isn't compatible with inheritance. +# The inheritance model needs to be reworked in order to support overriding args/ +# kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the +# constructor below and apply_defaults will no longer throw an exception. super(SqlSensor, self).__init__(*args, **kwargs) def poke(self, context): if self.first_poke: -self.first_poke = False +self.first_poke = False if '.' in self.table: self.schema, self.table = self.table.split('.') self.sql = """ @@ -301,9 +334,9 @@ LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE -B0.TBL_NAME = '{self.table}' AND -C0.NAME = '{self.schema}' AND -A0.PART_NAME = '{self.partition_name}'; +B0.TBL_NAME = '{self.table}' AND +C0.NAME = '{self.schema}' AND +A0.PART_NAME = '{self.partition_name}'; """.format(self=self) return super(MetastorePartitionSensor, self).poke(context) @@ -338,13 +371,13 @@ self,
[02/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/searchindex.js -- diff --git a/searchindex.js b/searchindex.js index 3d2f62d..625cb26 100644 --- a/searchindex.js +++ b/searchindex.js @@ -1 +1 @@ -Search.setIndex({envversion:47,filenames:["cli","code","concepts","configuration","faq","index","installation","license","plugins","profiling","project","scheduler","security","start","tutorial","ui"],objects:{"airflow.contrib":{hooks:[1,1,0,"-"],operators:[1,1,0,"-"]},"airflow.contrib.executors.mesos_executor":{MesosExecutor:[1,0,1,""]},"airflow.contrib.hooks":{BigQueryHook:[1,0,1,""],CloudantHook:[1,0,1,""],FTPHook:[1,0,1,""],GoogleCloudStorageHook:[1,0,1,""],SSHHook:[1,0,1,""],VerticaHook:[1,0,1,""]},"airflow.contrib.hooks.BigQueryHook":{get_conn:[1,2,1,""],get_pandas_df:[1,2,1,""],get_service:[1,2,1,""],insert_rows:[1,2,1,""]},"airflow.contrib.hooks.CloudantHook":{db:[1,2,1,""]},"airflow.contrib.hooks.FTPHook":{close_conn:[1,2,1,""],create_directory:[1,2,1,""],delete_directory:[1,2,1,""],delete_file:[1,2,1,""],describe_directory:[1,2,1,""],get_conn:[1,2,1,""],list_directory:[1,2,1,""],rename:[1,2,1,""],retrieve_file:[1,2,1,""],store_file:[1,2,1,""]},"airflow.contrib.hooks.Google CloudStorageHook":{download:[1,2,1,""],get_conn:[1,2,1,""],upload:[1,2,1,""]},"airflow.contrib.hooks.SSHHook":{Popen:[1,2,1,""],check_output:[1,2,1,""],tunnel:[1,2,1,""]},"airflow.contrib.hooks.VerticaHook":{get_conn:[1,2,1,""]},"airflow.contrib.hooks.gcs_hook":{GoogleCloudStorageHook:[1,0,1,""]},"airflow.contrib.operators":{QuboleOperator:[1,0,1,""],SSHExecuteOperator:[1,0,1,""],VerticaOperator:[1,0,1,""],VerticaToHiveTransfer:[1,0,1,""]},"airflow.contrib.operators.bigquery_operator":{BigQueryOperator:[1,0,1,""]},"airflow.contrib.operators.bigquery_to_gcs":{BigQueryToCloudStorageOperator:[1,0,1,""]},"airflow.contrib.operators.gcs_download_operator":{GoogleCloudStorageDownloadOperator:[1,0,1,""]},"airflow.contrib.operators.hipchat_operator":{HipChatAPIOperator:[1,0,1,""],HipChatAPISendRoomNotificationOperator:[1,0,1,""]},"airflow.executors":{CeleryExecutor:[1,0,1,""],LocalExecutor:[1,0,1,""],SequentialExecutor:[1,0,1,""]},"airflow.hooks":{DbApiHook:[1,0,1,""],DruidHook:[1,0,1,""],Hi veCliHook:[1,0,1,""],HiveMetastoreHook:[1,0,1,""],HiveServer2Hook:[1,0,1,""],HttpHook:[1,0,1,""],MsSqlHook:[1,0,1,""],MySqlHook:[1,0,1,""],PostgresHook:[1,0,1,""],PrestoHook:[1,0,1,""],S3Hook:[1,0,1,""],SqliteHook:[1,0,1,""],WebHDFSHook:[1,0,1,""]},"airflow.hooks.DbApiHook":{bulk_dump:[1,2,1,""],bulk_load:[1,2,1,""],get_conn:[1,2,1,""],get_cursor:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],insert_rows:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.DruidHook":{construct_ingest_query:[1,2,1,""],get_conn:[1,2,1,""],load_from_hdfs:[1,2,1,""]},"airflow.hooks.HiveCliHook":{load_file:[1,2,1,""],run_cli:[1,2,1,""],test_hql:[1,2,1,""]},"airflow.hooks.HiveMetastoreHook":{check_for_named_partition:[1,2,1,""],check_for_partition:[1,2,1,""],get_databases:[1,2,1,""],get_metastore_client:[1,2,1,""],get_partitions:[1,2,1,""],get_table:[1,2,1,""],get_tables:[1,2,1,""],max_partition:[1,2,1,""],table_exists:[1,2,1,""]},"airflow.hooks.HiveServer2Hook":{get_pandas_df:[1,2, 1,""],get_records:[1,2,1,""]},"airflow.hooks.HttpHook":{get_conn:[1,2,1,""],run:[1,2,1,""],run_and_check:[1,2,1,""]},"airflow.hooks.MsSqlHook":{get_conn:[1,2,1,""]},"airflow.hooks.MySqlHook":{bulk_load:[1,2,1,""],get_conn:[1,2,1,""]},"airflow.hooks.PrestoHook":{get_conn:[1,2,1,""],get_first:[1,2,1,""],get_pandas_df:[1,2,1,""],get_records:[1,2,1,""],run:[1,2,1,""]},"airflow.hooks.S3Hook":{check_for_bucket:[1,2,1,""],check_for_key:[1,2,1,""],check_for_prefix:[1,2,1,""],check_for_wildcard_key:[1,2,1,""],get_bucket:[1,2,1,""],get_conn:[1,2,1,""],get_key:[1,2,1,""],get_wildcard_key:[1,2,1,""],list_keys:[1,2,1,""],list_prefixes:[1,2,1,""],load_file:[1,2,1,""],load_string:[1,2,1,""]},"airflow.hooks.SqliteHook":{get_conn:[1,2,1,""]},"airflow.hooks.WebHDFSHook":{check_for_path:[1,2,1,""],get_conn:[1,2,1,""],load_file:[1,2,1,""]},"airflow.macros":{ds_add:[1,3,1,""],ds_format:[1,3,1,""],hive:[1,1,0,"-"],random:[1,3,1,""]},"airflow.macros.hive":{closest_ds_partition:[1,3,1,""],max_partition:[1, 3,1,""]},"airflow.models":{BaseOperator:[1,0,1,""],Connection:[1,0,1,""],DAG:[1,0,1,""],DagBag:[1,0,1,""],TaskInstance:[1,0,1,""]},"airflow.models.BaseOperator":{clear:[1,2,1,""],dag:[1,4,1,""],detect_downstream_cycle:[1,2,1,""],downstream_list:[1,4,1,""],execute:[1,2,1,""],get_direct_relatives:[1,2,1,""],get_flat_relatives:[1,2,1,""],get_task_instances:[1,2,1,""],has_dag:[1,2,1,""],on_kill:[1,2,1,""],post_execute:[1,2,1,""],pre_execute:[1,2,1,""],prepare_template:[1,2,1,""],render_template:[1,2,1,""],render_template_from_field:[1,2,1,""],run:[1,2,1,""],schedule_interval:[1,4,1,""],set_downstream:[1,2,1,""],set_upstream:[1,2,1,""],upstream_list:[1,4,1,""],xcom_pull:[1,2,1,""],xcom_push:[1,2,1,""]},
[01/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
Repository: incubator-airflow-site Updated Branches: refs/heads/asf-site 4af0850c3 -> 5e5740122 http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/security.html -- diff --git a/security.html b/security.html index 25ada3c..d0156b7 100644 --- a/security.html +++ b/security.html @@ -30,8 +30,11 @@ + + - + @@ -41,6 +44,7 @@ + @@ -102,12 +106,27 @@ Limitations Enabling kerberos Using kerberos authentication -GitHub Enterprise (GHE) Authentication -Setting up GHE Authentication +OAuth Authentication +GitHub Enterprise (GHE) Authentication +Setting up GHE Authentication + + +Google Authentication +Setting up Google Authentication + + + + +SSL +Impersonation + + +Experimental Rest API +Integration FAQ API Reference @@ -122,8 +141,10 @@ - -Airflow + + + Airflow + @@ -132,23 +153,40 @@ - + + + + + + + + + + + + -Docs » - -Security + + Docs » + + Security + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -156,13 +194,13 @@ Security¶ - -Web Authentication¶ By default, all gates are opened. An easy way to restrict access to the web application is to do it at the network level, or by using SSH tunnels. It is however possible to switch on authentication by either using one of the supplied backends or create your own. + +Web Authentication¶ Password¶ One of the simplest mechanisms for authentication is requiring users to specify a password before logging in. @@ -331,6 +369,9 @@ section of the connection. For the login user specify the following as extra: + + +OAuth Authentication¶ GitHub Enterprise (GHE) Authentication¶ The GitHub Enterprise authentication backend can be used to authenticate users @@ -348,12 +389,11 @@ your GHE installation will be able to login to Airflow. client_id = oauth_key_from_github_enterprise client_secret = oauth_secret_from_github_enterprise oauth_callback_route = /example/ghe_oauth/callback -allowed_teams = example_team_1, example_team_2 +allowed_teams = 1, 345, 23 - -Setting up GHE Authentication¶ +Setting up GHE Authentication¶ An application must be setup in GHE before you can use the GHE authentication backend. In order to setup an application: @@ -367,19 +407,87 @@ backend. In order to setup an application: + +Google Authentication¶ +The Google authentication backend can be used to authenticate users +against Google using OAuth2. You must specify a domain to restrict login +to only members of that domain. +[webserver] +authenticate = True +auth_backend = airflow.contrib.auth.backends.google_auth + +[google] +client_id = google_client_id +client_secret = google_client_secret +oauth_callback_route = /oauth2callback +domain = example.com + + + +Setting up Google Authentication¶ +An application must be setup in the Google API Console before you can use the Google authentication +backend. In order to setup an application: + +Navigate to https://console.developers.google.com/apis/";>https://console.developers.google.com/apis/ +Select ‘Credentials’ from the left hand nav +Click ‘Create credentials’ and choose ‘OAuth client ID’ +Choose ‘Web application’ +Fill in the required information (the ‘Authorized redirect URIs’ must be fully qualifed e.g. http://airflow.example.com/oauth2callback";>http://airflow.example.com/oauth2callback) +Click ‘Create’ +Copy ‘Client ID’, ‘Client Secret’, and your redirect URI to your airflow.cfg according to the above example + + + + + +SSL¶ +SSL can be enabled by providing a certificate and key. Once enabled, be sure to use +“https://";>https://” in your browser. +[webserver] +web_server_ssl_cert =+web_server_ssl_key = + + +Enabling SSL will not automatically change the web server port. If you want to use the +standard port 443, you’ll need to configure that too. Be aware that super user privileges +(or cap_net_bind_service on Linux) are required to listen on port 443. +# Optionally, set the server to listen on the standard SSL port. +web_server_port = 443 +base_url = http:// :443 + + + +Impersonation¶ +Airflow has the ability to impersonate a unix user while running task +instances based on the task’s run_as_user parameter, which takes a user’s name. +NOTE For impersonations to work, Airflow must be run with sudo as subtasks are run +with sudo -u and permissions of files are changed. Furthermore, the unix user needs to +exist on the worker. Here is what a simple sudoers file entry could look like to achieve +this, ass
[22/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
Latest docs version as of 1.8.x Project: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/commit/5e574012 Tree: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/tree/5e574012 Diff: http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/diff/5e574012 Branch: refs/heads/asf-site Commit: 5e5740122ed33a22a30047e75e6ca4c7da3961b4 Parents: 9c75ee9 Author: Maxime Beauchemin Authored: Mon Mar 6 08:43:25 2017 -0800 Committer: Maxime Beauchemin Committed: Mon Mar 6 08:43:25 2017 -0800 -- _images/latest_only_with_trigger.png | Bin 0 -> 40034 bytes _sources/api.rst.txt |43 + _sources/cli.rst.txt |11 + _sources/code.rst.txt| 255 + _sources/concepts.rst.txt| 833 +++ _sources/configuration.rst.txt | 284 + _sources/faq.rst.txt | 147 + _sources/index.rst.txt |89 + _sources/installation.rst.txt|90 + _sources/integration.rst.txt | 246 + _sources/license.rst.txt | 211 + _sources/plugins.rst.txt | 144 + _sources/profiling.rst.txt |39 + _sources/project.rst.txt |49 + _sources/scheduler.rst.txt | 153 + _sources/security.rst.txt| 334 + _sources/start.rst.txt |49 + _sources/tutorial.rst.txt| 429 ++ _sources/ui.rst.txt | 102 + _static/fonts/Inconsolata.ttf| Bin 0 -> 63184 bytes _static/jquery-3.1.0.js | 10074 api.html | 279 + integration.html | 424 ++ 23 files changed, 14285 insertions(+) -- http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_images/latest_only_with_trigger.png -- diff --git a/_images/latest_only_with_trigger.png b/_images/latest_only_with_trigger.png new file mode 100644 index 000..629adfa Binary files /dev/null and b/_images/latest_only_with_trigger.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/api.rst.txt -- diff --git a/_sources/api.rst.txt b/_sources/api.rst.txt new file mode 100644 index 000..eef671c --- /dev/null +++ b/_sources/api.rst.txt @@ -0,0 +1,43 @@ +Experimental Rest API += + +Airflow exposes an experimental Rest API. It is available through the webserver. Endpoints are +available at /api/experimental/. Please note that we expect the endpoint definitions to change. + +Endpoints +- + +This is a place holder until the swagger definitions are active + +* /api/experimental/dags//tasks/ returns info for a task (GET). +* /api/experimental/dags//dag_runs creates a dag_run for a given dag id (POST). + +CLI +- + +For some functions the cli can use the API. To configure the CLI to use the API when available +configure as follows: + +.. code-block:: bash + +[cli] +api_client = airflow.api.client.json_client +endpoint_url = http://: + + +Authentication +-- + +Only Kerberos authentication is currently supported for the API. To enable this set the following +in the configuration: + +.. code-block:: bash + +[api] +auth_backend = airflow.api.auth.backend.default + +[kerberos] +keytab = + +The Kerberos service is configured as `airflow/fully.qualified.domainname@REALM`. Make sure this +principal exists in the keytab file. http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/cli.rst.txt -- diff --git a/_sources/cli.rst.txt b/_sources/cli.rst.txt new file mode 100644 index 000..f05cbfb --- /dev/null +++ b/_sources/cli.rst.txt @@ -0,0 +1,11 @@ +Command Line Interface +== + +Airflow has a very rich command line interface that allows for +many types of operation on a DAG, starting services, and supporting +development and testing. + +.. argparse:: + :module: airflow.bin.cli + :func: get_parser + :prog: airflow http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/code.rst.txt -- diff --git a/_sources/code.rst.txt b/_sources/code.rst.txt new file mode 100644 index 000..fabe6db --- /dev/null +++ b/_sources/code.rst.txt @@ -0,0 +1,255 @@ +API Reference += + +Operators +- +Operators allow for generation of certain types of tasks that become nodes in +the DAG when instantiated. All operators derive from BaseOperator and +inherit many at
[21/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_sources/license.rst.txt -- diff --git a/_sources/license.rst.txt b/_sources/license.rst.txt new file mode 100644 index 000..9da26c0 --- /dev/null +++ b/_sources/license.rst.txt @@ -0,0 +1,211 @@ +License +=== + +.. image:: img/apache.jpg +:width: 150 + +:: + +Apache License +Version 2.0, January 2004 +http://www.apache.org/licenses/ + +TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + +1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + +2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + +3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to
[05/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/concepts.html -- diff --git a/concepts.html b/concepts.html index 15b2b4d..1ed51a5 100644 --- a/concepts.html +++ b/concepts.html @@ -30,6 +30,9 @@ + + @@ -41,6 +44,7 @@ + @@ -114,6 +118,7 @@ SubDAGs SLAs Trigger Rules +Latest Run Only Zombies & Undeads Cluster Policy Documentation & Notes @@ -128,6 +133,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -142,8 +149,10 @@ - -Airflow + + + Airflow + @@ -152,23 +161,40 @@ - + + + + + + + + + + + + -Docs » - -Concepts + + Docs » + + Concepts + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -341,8 +367,7 @@ object is always returned. For example: We can put this all together to build a simple pipeline: with DAG('my_dag', start_date=datetime(2016, 1, 1)) as dag: ( -dag ->> DummyOperator(task_id='dummy_1') +DummyOperator(task_id='dummy_1') >> BashOperator( task_id='bash_1', bash_command='echo "HELLO!"') @@ -452,10 +477,11 @@ variables from the operating system. The environment variable needs to be prefixed with AIRFLOW_CONN_ to be considered a connection. When referencing the connection in the Airflow pipeline, the conn_id should be the name of the variable without the prefix. For example, if the conn_id -is named POSTGRES_MASTER the environment variable should be named -AIRFLOW_CONN_POSTGRES_MASTER. Airflow assumes the value returned -from the environment variable to be in a URI format -(e.g. postgres://user:password@localhost:5432/master). +is named postgres_master the environment variable should be named +AIRFLOW_CONN_POSTGRES_MASTER (note that the environment variable must be +all uppercase). Airflow assumes the value returned from the environment +variable to be in a URI format (e.g. +postgres://user:password@localhost:5432/master or s3://accesskey:secretkey@S3). Queues¶ @@ -674,6 +700,71 @@ while creating tasks: that, when set to True, keeps a task from getting triggered if the previous schedule for the task hasn’t succeeded. + +Latest Run Only¶ +Standard workflow behavior involves running a series of tasks for a +particular date/time range. Some workflows, however, perform tasks that +are independent of run time but need to be run on a schedule, much like a +standard cron job. In these cases, backfills or running jobs missed during +a pause just wastes CPU cycles. +For situations like this, you can use the LatestOnlyOperator to skip +tasks that are not being run during the most recent scheduled run for a +DAG. The LatestOnlyOperator skips all immediate downstream tasks, and +itself, if the time right now is not between its execution_time and the +next scheduled execution_time. +One must be aware of the interaction between skipped tasks and trigger +rules. Skipped tasks will cascade through trigger rules all_success +and all_failed but not all_done, one_failed, one_success, +and dummy. If you would like to use the LatestOnlyOperator with +trigger rules that do not cascade skips, you will need to ensure that the +LatestOnlyOperator is directly upstream of the task you would like +to skip. +It is possible, through use of trigger rules to mix tasks that should run +in the typical date/time dependent mode and those using the +LatestOnlyOperator. +For example, consider the following dag: +#dags/latest_only_with_trigger.py +import datetime as dt + +from airflow.models import DAG +from airflow.operators.dummy_operator import DummyOperator +from airflow.operators.latest_only_operator import LatestOnlyOperator +from airflow.utils.trigger_rule import TriggerRule + + +dag = DAG( +dag_id='latest_only_with_trigger', +schedule_interval=dt.timedelta(hours=4), +start_date=dt.datetime(2016, 9, 20), +) + +latest_only = LatestOnlyOperator(task_id='latest_only', dag=dag) + +task1 = DummyOperator(task_id='task1', dag=dag) +task1.set_upstream(latest_only) + +task2 = DummyOperator(task_id='task2', dag=dag) + +task3 = DummyOperator(task_id='task3', dag=dag) +task3.set_upstream([task1, task2]) + +task4 = DummyOperator(task_id='task4', dag=dag, + trigger_rule=TriggerRule.ALL_DONE) +task4.set_upstream([task1, task2]) + + +In the case of this dag, the latest_only task will show up as skipped +for all runs except the latest run. task1 is directly downstream of +latest_only and will also skip for all runs except the latest. +task2 is entirely independent of latest_only an
[13/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/sensors.html -- diff --git a/_modules/sensors.html b/_modules/sensors.html index 6567e38..f2b1cfe 100644 --- a/_modules/sensors.html +++ b/_modules/sensors.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,19 +126,36 @@ + + + + + + + + + + + -Docs » - + + Docs » + Module code » - -sensors + + sensors + + - + + + + http://schema.org/Article";> @@ -151,24 +176,27 @@ # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import print_function -from future import standard_library +from __future__ import print_function +from future import standard_library standard_library.install_aliases() -from builtins import str -from past.builtins import basestring +from builtins import str +from past.builtins import basestring -from datetime import datetime +from datetime import datetime import logging -from urllib.parse import urlparse -from time import sleep +from urllib.parse import urlparse +from time import sleep +import re +import sys import airflow -from airflow import hooks, settings -from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException -from airflow.models import BaseOperator, TaskInstance, Connection as DB -from airflow.hooks.base_hook import BaseHook -from airflow.utils.state import State -from airflow.utils.decorators import apply_defaults +from airflow import hooks, settings +from airflow.exceptions import AirflowException, AirflowSensorTimeout, AirflowSkipException +from airflow.models import BaseOperator, TaskInstance +from airflow.hooks.base_hook import BaseHook +from airflow.hooks.hdfs_hook import HDFSHook +from airflow.utils.state import State +from airflow.utils.decorators import apply_defaults class BaseSensorOperator(BaseOperator): @@ -193,7 +221,7 @@ self, poke_interval=60, timeout=60*60*24*7, -soft_fail=False, +soft_fail=False, *args, **kwargs): super(BaseSensorOperator, self).__init__(*args, **kwargs) self.poke_interval = poke_interval @@ -245,13 +273,13 @@ logging.info('Poking: ' + self.sql) records = hook.get_records(self.sql) if not records: -return False +return False else: if str(records[0][0]) in ('0', '',): -return False +return False else: -return True -print(records[0][0]) +return True +print(records[0][0]) [docs]class MetastorePartitionSensor(SqlSensor): @@ -286,13 +314,18 @@ self.partition_name = partition_name self.table = table self.schema = schema -self.first_poke = True +self.first_poke = True self.conn_id = mysql_conn_id +# TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor. +# The problem is the way apply_defaults works isn't compatible with inheritance. +# The inheritance model needs to be reworked in order to support overriding args/ +# kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the +# constructor below and apply_defaults will no longer throw an exception. super(SqlSensor, self).__init__(*args, **kwargs) def poke(self, context): if self.first_poke: -self.first_poke = False +self.first_poke = False if '.' in self.table: self.schema, self.table = self.table.split('.') self.sql = """ @@ -301,9 +334,9 @@ LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID WHERE -B0.TBL_NAME = '{self.table}' AND -C0.NAME = '{self.schema}' AND -A0.PART_NAME = '{self.partition_name}'; +B0.TBL_NAME = '{self.table}' AND +C0.NAME = '{self.schema}' AND +A0.PART_NAME = '{self.partition_name}'; """.format(self=self) return super(MetastorePartitionSensor, self).poke(context) @@ -338,13 +371,13 @@ self, external_dag_id, external_task_id, -allowed_states=None, -execution_delta=None, -
[15/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/dbapi_hook.html -- diff --git a/_modules/dbapi_hook.html b/_modules/dbapi_hook.html index 604..ae37fd3 100644 --- a/_modules/dbapi_hook.html +++ b/_modules/dbapi_hook.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,19 +126,36 @@ + + + + + + + + + + + -Docs » - + + Docs » + Module code » - -dbapi_hook + + dbapi_hook + + - + + + + http://schema.org/Article";> @@ -151,15 +176,17 @@ # See the License for the specific language governing permissions and # limitations under the License. -from builtins import str -from past.builtins import basestring -from datetime import datetime +from builtins import str +from past.builtins import basestring +from datetime import datetime import numpy import logging import sys -from airflow.hooks.base_hook import BaseHook -from airflow.exceptions import AirflowException +from sqlalchemy import create_engine + +from airflow.hooks.base_hook import BaseHook +from airflow.exceptions import AirflowException [docs]class DbApiHook(BaseHook): @@ -167,13 +194,13 @@ Abstract base class for sql hooks. """ # Override to provide the connection name. -conn_name_attr = None +conn_name_attr = None # Override to have a default connection id for a particular dbHook default_conn_name = 'default_conn_id' # Override if this db supports autocommit. -supports_autocommit = False +supports_autocommit = False # Override with the object that exposes the connect method -connector = None +connector = None def __init__(self, *args, **kwargs): if not self.conn_name_attr: @@ -195,7 +222,23 @@ username=db.login, schema=db.schema) -[docs]def get_pandas_df(self, sql, parameters=None): +def get_uri(self): +conn = self.get_connection(getattr(self, self.conn_name_attr)) +login = '' +if conn.login: +login = '{conn.login}:{conn.password}@'.format(conn=conn) +host = conn.host +if conn.port is not None: +host += ':{port}'.format(port=conn.port) +return '{conn.conn_type}://{login}{host}/{conn.schema}'.format( +conn=conn, login=login, host=host) + +def get_sqlalchemy_engine(self, engine_kwargs=None): +if engine_kwargs is None: +engine_kwargs = {} +return create_engine(self.get_uri(), **engine_kwargs) + +[docs]def get_pandas_df(self, sql, parameters=None): """ Executes the sql and returns a pandas dataframe @@ -207,13 +250,13 @@ """ if sys.version_info[0] < 3: sql = sql.encode('utf-8') -import pandas.io.sql as psql +import pandas.io.sql as psql conn = self.get_conn() df = psql.read_sql(sql, con=conn, params=parameters) conn.close() return df -[docs]def get_records(self, sql, parameters=None): +[docs]def get_records(self, sql, parameters=None): """ Executes the sql and returns a set of records. @@ -227,7 +270,7 @@ sql = sql.encode('utf-8') conn = self.get_conn() cur = self.get_cursor() -if parameters is not None: +if parameters is not None: cur.execute(sql, parameters) else: cur.execute(sql) @@ -236,7 +279,7 @@ conn.close() return rows -[docs]def get_first(self, sql, parameters=None): +[docs]def get_first(self, sql, parameters=None): """ Executes the sql and returns the first resulting row. @@ -250,7 +293,7 @@ sql = sql.encode('utf-8') conn = self.get_conn() cur = conn.cursor() -if parameters is not None: +if parameters is not None: cur.execute(sql, parameters) else: cur.execute(sql) @@ -259,7 +302,7 @@ conn.close() return rows -[docs]def run(self, sql, autocommit=False, parameters=None): +[docs]def run(self, sql, autocommit=False, parameters=None): """ Runs a command or a list of commands. Pass a list of sql statements to the sql parameter to get them to execute @@ -275,7 +318,7 @@ :type parameters: mapping or iterable """ conn = self.get_conn() -if isinstance(sql, basestring): +if isinstance(sql, basestri
[08/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.woff -- diff --git a/_static/fonts/fontawesome-webfont.woff b/_static/fonts/fontawesome-webfont.woff index 628b6a5..6e7483c 100644 Binary files a/_static/fonts/fontawesome-webfont.woff and b/_static/fonts/fontawesome-webfont.woff differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/jquery.js -- diff --git a/_static/jquery.js b/_static/jquery.js index ab28a24..f6a6a99 100644 --- a/_static/jquery.js +++ b/_static/jquery.js @@ -1,4 +1,4 @@ -/*! jQuery v1.11.1 | (c) 2005, 2014 jQuery Foundation, Inc. | jquery.org/license */ -!function(a,b){"object"==typeof module&&"object"==typeof module.exports?module.exports=a.document?b(a,!0):function(a){if(!a.document)throw new Error("jQuery requires a window with a document");return b(a)}:b(a)}("undefined"!=typeof window?window:this,function(a,b){var c=[],d=c.slice,e=c.concat,f=c.push,g=c.indexOf,h={},i=h.toString,j=h.hasOwnProperty,k={},l="1.11.1",m=function(a,b){return new m.fn.init(a,b)},n=/^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g,o=/^-ms-/,p=/-([\da-z])/gi,q=function(a,b){return b.toUpperCase()};m.fn=m.prototype={jquery:l,constructor:m,selector:"",length:0,toArray:function(){return d.call(this)},get:function(a){return null!=a?0>a?this[a+this.length]:this[a]:d.call(this)},pushStack:function(a){var b=m.merge(this.constructor(),a);return b.prevObject=this,b.context=this.context,b},each:function(a,b){return m.each(this,a,b)},map:function(a){return this.pushStack(m.map(this,function(b,c){return a.call(b,c,b)}))},slice:function(){return this.pushStack(d.apply(this,argumen ts))},first:function(){return this.eq(0)},last:function(){return this.eq(-1)},eq:function(a){var b=this.length,c=+a+(0>a?b:0);return this.pushStack(c>=0&&b>c?[this[c]]:[])},end:function(){return this.prevObject||this.constructor(null)},push:f,sort:c.sort,splice:c.splice},m.extend=m.fn.extend=function(){var a,b,c,d,e,f,g=arguments[0]||{},h=1,i=arguments.length,j=!1;for("boolean"==typeof g&&(j=g,g=arguments[h]||{},h++),"object"==typeof g||m.isFunction(g)||(g={}),h===i&&(g=this,h--);i>h;h++)if(null!=(e=arguments[h]))for(d in e)a=g[d],c=e[d],g!==c&&(j&&c&&(m.isPlainObject(c)||(b=m.isArray(c)))?(b?(b=!1,f=a&&m.isArray(a)?a:[]):f=a&&m.isPlainObject(a)?a:{},g[d]=m.extend(j,f,c)):void 0!==c&&(g[d]=c));return g},m.extend({expando:"jQuery"+(l+Math.random()).replace(/\D/g,""),isReady:!0,error:function(a){throw new Error(a)},noop:function(){},isFunction:function(a){return"function"===m.type(a)},isArray:Array.isArray||function(a){return"array"===m.type(a)},isWindow:function(a){return null!=a&&a= =a.window},isNumeric:function(a){return!m.isArray(a)&&a-parseFloat(a)>=0},isEmptyObject:function(a){var b;for(b in a)return!1;return!0},isPlainObject:function(a){var b;if(!a||"object"!==m.type(a)||a.nodeType||m.isWindow(a))return!1;try{if(a.constructor&&!j.call(a,"constructor")&&!j.call(a.constructor.prototype,"isPrototypeOf"))return!1}catch(c){return!1}if(k.ownLast)for(b in a)return j.call(a,b);for(b in a);return void 0===b||j.call(a,b)},type:function(a){return null==a?a+"":"object"==typeof a||"function"==typeof a?h[i.call(a)]||"object":typeof a},globalEval:function(b){b&&m.trim(b)&&(a.execScript||function(b){a.eval.call(a,b)})(b)},camelCase:function(a){return a.replace(o,"ms-").replace(p,q)},nodeName:function(a,b){return a.nodeName&&a.nodeName.toLowerCase()===b.toLowerCase()},each:function(a,b,c){var d,e=0,f=a.length,g=r(a);if(c){if(g){for(;f>e;e++)if(d=b.apply(a[e],c),d===!1)break}else for(e in a)if(d=b.apply(a[e],c),d===!1)break}else if(g){for(;f>e;e++)if(d=b.call(a[e],e,a[e]),d ===!1)break}else for(e in a)if(d=b.call(a[e],e,a[e]),d===!1)break;return a},trim:function(a){return null==a?"":(a+"").replace(n,"")},makeArray:function(a,b){var c=b||[];return null!=a&&(r(Object(a))?m.merge(c,"string"==typeof a?[a]:a):f.call(c,a)),c},inArray:function(a,b,c){var d;if(b){if(g)return g.call(b,a,c);for(d=b.length,c=c?0>c?Math.max(0,d+c):c:0;d>c;c++)if(c in b&&b[c]===a)return c}return-1},merge:function(a,b){var c=+b.length,d=0,e=a.length;while(c>d)a[e++]=b[d++];if(c!==c)while(void 0!==b[d])a[e++]=b[d++];return a.length=e,a},grep:function(a,b,c){for(var d,e=[],f=0,g=a.length,h=!c;g>f;f++)d=!b(a[f],f),d!==h&&e.push(a[f]);return e},map:function(a,b,c){var d,f=0,g=a.length,h=r(a),i=[];if(h)for(;g>f;f++)d=b(a[f],f,c),null!=d&&i.push(d);else for(f in a)d=b(a[f],f,c),null!=d&&i.push(d);return e.apply([],i)},guid:1,proxy:function(a,b){var c,e,f;return"string"==typeof b&&(f=a[b],b=a,a=f),m.isFunction(a)?(c=d.call(arguments,2),e=function(){return a.apply(b||this,c.concat(d.call(ar guments)))},e.guid=a.guid=a.guid||m.guid++,e):void 0},now:function(){return+new Date},support:k}),m.each("Boolean Number String F
[20/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/5e574012/_static/jquery-3.1.0.js -- diff --git a/_static/jquery-3.1.0.js b/_static/jquery-3.1.0.js new file mode 100644 index 000..f2fc274 --- /dev/null +++ b/_static/jquery-3.1.0.js @@ -0,0 +1,10074 @@ +/*eslint-disable no-unused-vars*/ +/*! + * jQuery JavaScript Library v3.1.0 + * https://jquery.com/ + * + * Includes Sizzle.js + * https://sizzlejs.com/ + * + * Copyright jQuery Foundation and other contributors + * Released under the MIT license + * https://jquery.org/license + * + * Date: 2016-07-07T21:44Z + */ +( function( global, factory ) { + + "use strict"; + + if ( typeof module === "object" && typeof module.exports === "object" ) { + + // For CommonJS and CommonJS-like environments where a proper `window` + // is present, execute the factory and get jQuery. + // For environments that do not have a `window` with a `document` + // (such as Node.js), expose a factory as module.exports. + // This accentuates the need for the creation of a real `window`. + // e.g. var jQuery = require("jquery")(window); + // See ticket #14549 for more info. + module.exports = global.document ? + factory( global, true ) : + function( w ) { + if ( !w.document ) { + throw new Error( "jQuery requires a window with a document" ); + } + return factory( w ); + }; + } else { + factory( global ); + } + +// Pass this if window is not defined yet +} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { + +// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 +// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode +// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common +// enough that all such attempts are guarded in a try block. +"use strict"; + +var arr = []; + +var document = window.document; + +var getProto = Object.getPrototypeOf; + +var slice = arr.slice; + +var concat = arr.concat; + +var push = arr.push; + +var indexOf = arr.indexOf; + +var class2type = {}; + +var toString = class2type.toString; + +var hasOwn = class2type.hasOwnProperty; + +var fnToString = hasOwn.toString; + +var ObjectFunctionString = fnToString.call( Object ); + +var support = {}; + + + + function DOMEval( code, doc ) { + doc = doc || document; + + var script = doc.createElement( "script" ); + + script.text = code; + doc.head.appendChild( script ).parentNode.removeChild( script ); + } +/* global Symbol */ +// Defining this global in .eslintrc would create a danger of using the global +// unguarded in another place, it seems safer to define global only for this module + + + +var + version = "3.1.0", + + // Define a local copy of jQuery + jQuery = function( selector, context ) { + + // The jQuery object is actually just the init constructor 'enhanced' + // Need init if jQuery is called (just allow error to be thrown if not included) + return new jQuery.fn.init( selector, context ); + }, + + // Support: Android <=4.0 only + // Make sure we trim BOM and NBSP + rtrim = /^[\s\uFEFF\xA0]+|[\s\uFEFF\xA0]+$/g, + + // Matches dashed string for camelizing + rmsPrefix = /^-ms-/, + rdashAlpha = /-([a-z])/g, + + // Used by jQuery.camelCase as callback to replace() + fcamelCase = function( all, letter ) { + return letter.toUpperCase(); + }; + +jQuery.fn = jQuery.prototype = { + + // The current version of jQuery being used + jquery: version, + + constructor: jQuery, + + // The default length of a jQuery object is 0 + length: 0, + + toArray: function() { + return slice.call( this ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + return num != null ? + + // Return just the one element from the set + ( num < 0 ? this[ num + this.length ] : this[ num ] ) : + + // Return all the elements in a clean array + slice.call( this ); + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems ) { + + // Build a new jQuery matched element set + var ret = jQuery.merge( this.const
[11/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/css/theme.css -- diff --git a/_static/css/theme.css b/_static/css/theme.css index 7be9339..c1631d8 100644 --- a/_static/css/theme.css +++ b/_static/css/theme.css @@ -1,5 +1,5 @@ *{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}audio:not([controls]){display:none}[hidden]{display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:hover,a:active{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:bold}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;color:#000;text-decoration:none}mark{background:#ff0;color:#000;font-style:italic;font-weight:bold}pre,code,.rst-content tt,.rst-content code,kbd,samp{font-family:monospace,serif;_font-family:"courier new",monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:before,q:after{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;ver tical-align:baseline}sup{top:-0.5em}sub{bottom:-0.25em}ul,ol,dl{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure{margin:0}form{margin:0}fieldset{border:0;margin:0;padding:0}label{cursor:pointer}legend{border:0;*margin-left:-7px;padding:0;white-space:normal}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type="button"],input[type="reset"],input[type="submit"]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type="checkbox"],input[type="radio"]{box-sizing:border-box;padding:0;*width:13px;*height:13px}input[type="search"]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}input[type="search"]::-webkit-search-decor ation,input[type="search"]::-webkit-search-cancel-button{-webkit-appearance:none}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}textarea{overflow:auto;vertical-align:top;resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:0.2em 0;background:#ccc;color:#000;padding:0.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none !important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{html,body,section{background:none !important}*{box-shadow:n one !important;text-shadow:none !important;filter:none !important;-ms-filter:none !important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="javascript:"]:after,a[href^="#"]:after{content:""}pre,blockquote{page-break-inside:avoid}thead{display:table-header-group}tr,img{page-break-inside:avoid}img{max-width:100% !important}@page{margin:0.5cm}p,h2,.rst-content .toctree-wrapper p.caption,h3{orphans:3;widows:3}h2,.rst-content .toctree-wrapper p.caption,h3{page-break-after:avoid}}.fa:before,.wy-menu-vertical li span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.rst-content .admonition-title:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content dl dt .headerlink:before,.rst-content p.caption .headerlink:before,.rst-con tent tt.download span:first-child:before,.rst-content code.download span:first-child:before,.icon:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-alert,.rst-content .note,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .warning,.rst-content .seealso,
[09/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.svg -- diff --git a/_static/fonts/fontawesome-webfont.svg b/_static/fonts/fontawesome-webfont.svg index d907b25..8b66187 100644 --- a/_static/fonts/fontawesome-webfont.svg +++ b/_static/fonts/fontawesome-webfont.svg @@ -147,14 +147,14 @@ - + - + @@ -169,7 +169,7 @@ - + @@ -178,7 +178,7 @@ - + @@ -219,8 +219,8 @@ - - + + @@ -275,7 +275,7 @@ - + @@ -362,8 +362,8 @@ - - + + @@ -399,7 +399,7 @@ - + @@ -410,9 +410,9 @@ - - - + + + @@ -438,7 +438,7 @@ - + @@ -454,12 +454,12 @@ - + - + @@ -483,13 +483,13 @@ - - + + - + @@ -513,8 +513,173 @@ - - + + + + + + + + + +http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/fonts/fontawesome-webfont.ttf -- diff --git a/_static/fonts/fontawesome-webfont.ttf b/_static/fonts/fontawesome-webfont.ttf index 96a3639..f221e50 100644 Binary files a/_static/fonts/fontawesome-webfont.ttf and b/_static/fonts/fontawesome-webfont.ttf differ
[12/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/ssh_hook.html -- diff --git a/_modules/ssh_hook.html b/_modules/ssh_hook.html index e094b62..8da4b35 100644 --- a/_modules/ssh_hook.html +++ b/_modules/ssh_hook.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,19 +126,36 @@ + + + + + + + + + + + -Docs » - + + Docs » + Module code » - -ssh_hook + + ssh_hook + + - + + + + http://schema.org/Article";> @@ -156,10 +181,10 @@ # # This is a port of Luigi's ssh implementation. All credits go there. import subprocess -from contextlib import contextmanager +from contextlib import contextmanager -from airflow.hooks.base_hook import BaseHook -from airflow.exceptions import AirflowException +from airflow.hooks.base_hook import BaseHook +from airflow.exceptions import AirflowException import logging @@ -191,11 +216,13 @@ """ def __init__(self, conn_id='ssh_default'): conn = self.get_connection(conn_id) -self.key_file = conn.extra_dejson.get('key_file', None) -self.connect_timeout = conn.extra_dejson.get('connect_timeout', None) -self.no_host_key_check = conn.extra_dejson.get('no_host_key_check', False) -self.tty = conn.extra_dejson.get('tty', False) -self.sshpass = conn.extra_dejson.get('sshpass', False) +self.key_file = conn.extra_dejson.get('key_file', None) +self.connect_timeout = conn.extra_dejson.get('connect_timeout', None) +self.tcp_keepalive = conn.extra_dejson.get('tcp_keepalive', False) +self.server_alive_interval = conn.extra_dejson.get('server_alive_interval', 60) +self.no_host_key_check = conn.extra_dejson.get('no_host_key_check', False) +self.tty = conn.extra_dejson.get('tty', False) +self.sshpass = conn.extra_dejson.get('sshpass', False) self.conn = conn def get_conn(self): @@ -203,7 +230,7 @@ def _host_ref(self): if self.conn.login: -return "{0}@{1}".format(self.conn.login, self.conn.host) +return "{0}@{1}".format(self.conn.login, self.conn.host) else: return self.conn.host @@ -218,7 +245,11 @@ connection_cmd += ["-p", str(self.conn.port)] if self.connect_timeout: -connection_cmd += ["-o", "ConnectionTimeout={}".format(self.connect_timeout)] +connection_cmd += ["-o", "ConnectionTimeout={}".format(self.connect_timeout)] + +if self.tcp_keepalive: +connection_cmd += ["-o", "TCPKeepAlive=yes"] +connection_cmd += ["-o", "ServerAliveInterval={}".format(self.server_alive_interval)] if self.no_host_key_check: connection_cmd += ["-o", "UserKnownHostsFile=/dev/null", @@ -231,7 +262,7 @@ connection_cmd += ["-t"] connection_cmd += cmd -logging.debug("SSH cmd: {} ".format(connection_cmd)) +logging.debug("SSH cmd: {} ".format(connection_cmd)) return connection_cmd @@ -259,13 +290,13 @@ if p.returncode != 0: # I like this better: RemoteCalledProcessError(p.returncode, cmd, self.host, output=output) -raise AirflowException("Cannot execute {} on {}. Error code is: {}. Output: {}, Stderr: {}".format( +raise AirflowException("Cannot execute {} on {}. Error code is: {}. Output: {}, Stderr: {}".format( cmd, self.conn.host, p.returncode, output, stderr)) return output @contextmanager -[docs]def tunnel(self, local_port, remote_port=None, remote_host="localhost"): +[docs]def tunnel(self, local_port, remote_port=None, remote_host="localhost"): """ Creates a tunnel between two hosts. Like ssh -L:host: . Remember to close() the returned "tunnel" object in order to clean up @@ -279,7 +310,7 @@ :type remote_host: str :return: """ -tunnel_host = "{0}:{1}:{2}".format(local_port, remote_host, remote_port) +tunnel_host = "{0}:{1}:{2}".format(local_port, remote_host, remote_port) proc = self.Popen(["-L", tunnel_host, "echo -n ready && cat"], stdin=subprocess.PIPE, stdout=subprocess.PIPE) @@ -287,11 +318,14 @@ assert ready == b"ready", "Did not get 'ready' from remote" yield proc.communicate() -assert proc.returncod
[03/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/plugins.html -- diff --git a/plugins.html b/plugins.html index 68c615d..be1cfbb 100644 --- a/plugins.html +++ b/plugins.html @@ -30,6 +30,9 @@ + + @@ -41,6 +44,7 @@ + @@ -97,6 +101,8 @@ Security +Experimental Rest API +Integration FAQ API Reference @@ -111,8 +117,10 @@ - -Airflow + + + Airflow + @@ -121,23 +129,40 @@ - + + + + + + + + + + + + -Docs » - -Plugins + + Docs » + + Plugins + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -226,18 +251,22 @@ definitions in Airflow. from airflow.models import BaseOperator from airflow.executors.base_executor import BaseExecutor -# Will show up under airflow.hooks.PluginHook +# Will show up under airflow.hooks.test_plugin.PluginHook class PluginHook(BaseHook): pass -# Will show up under airflow.operators.PluginOperator +# Will show up under airflow.operators.test_plugin.PluginOperator class PluginOperator(BaseOperator): pass -# Will show up under airflow.executors.PluginExecutor +# Will show up under airflow.executors.test_plugin.PluginExecutor class PluginExecutor(BaseExecutor): pass +# Will show up under airflow.macros.test_plugin.plugin_macro +def plugin_macro(): +pass + # Creating a flask admin BaseView class TestView(BaseView): @expose('/') @@ -262,10 +291,11 @@ definitions in Airflow. class AirflowTestPlugin(AirflowPlugin): name = "test_plugin" operators = [PluginOperator] -flask_blueprints = [bp] hooks = [PluginHook] executors = [PluginExecutor] +macros = [plugin_macro] admin_views = [v] +flask_blueprints = [bp] menu_links = [ml] @@ -274,15 +304,18 @@ definitions in Airflow. + + + -Next +Next - Previous + Previous @@ -315,7 +348,8 @@ definitions in Airflow. VERSION:'', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', -HAS_SOURCE: true +HAS_SOURCE: true, +SOURCELINK_SUFFIX: '.txt' }; http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/profiling.html -- diff --git a/profiling.html b/profiling.html index 795ec81..350c015 100644 --- a/profiling.html +++ b/profiling.html @@ -30,6 +30,9 @@ + + @@ -41,6 +44,7 @@ + @@ -99,6 +103,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -113,8 +119,10 @@ - -Airflow + + + Airflow + @@ -123,23 +131,40 @@ - + + + + + + + + + + + + -Docs » - -Data Profiling + + Docs » + + Data Profiling + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -181,15 +206,18 @@ directly in the URL. + + + -Next +Next - Previous + Previous @@ -222,7 +250,8 @@ directly in the URL. VERSION:'', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', -HAS_SOURCE: true +HAS_SOURCE: true, +SOURCELINK_SUFFIX: '.txt' }; http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/project.html -- diff --git a/project.html b/project.html index d2eb7f2..1bc09d0 100644 --- a/project.html +++ b/project.html @@ -30,6 +30,9 @@ + + @@ -41,6 +44,7 @@ + @@ -97,6 +101,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -111,8 +117,10 @@ - -Airflow + + + Airflow + @@ -121,23 +129,40 @@ - + + + + + + + + + + + + -Docs » - -Project + + Docs » + + Project +
[04/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/genindex.html -- diff --git a/genindex.html b/genindex.html index d4a80fb..76acd11 100644 --- a/genindex.html +++ b/genindex.html @@ -31,6 +31,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,17 +126,34 @@ + + + + + + + + + + + -Docs » - - + + Docs » + + + + - + + + + http://schema.org/Article";> @@ -153,1167 +178,536 @@ | N | O | P - | Q | R | S | T | U - | V | W | X A - - - add_task() (airflow.models.DAG method) - - - - add_tasks() (airflow.models.DAG method) - - - - airflow.contrib.hooks (module) - - - - airflow.contrib.operators (module) - - - - airflow.executors (module) - - - - airflow.hooks (module) - - - - - - airflow.macros (module) - - - - airflow.macros.hive (module) - - - - airflow.models (module) - - - - airflow.operators (module) - - - - are_dependencies_met() (airflow.models.TaskInstance method) - - - - are_dependents_done() (airflow.models.TaskInstance method) - - - + + add_task() (airflow.models.DAG method) + + add_tasks() (airflow.models.DAG method) + + airflow.contrib.hooks (module) + + airflow.contrib.operators (module) + + airflow.executors (module) + + airflow.hooks (module) + + + + airflow.macros (module) + + airflow.macros.hive (module) + + airflow.models (module) + + airflow.operators (module) + + are_dependencies_met() (airflow.models.TaskInstance method) + + are_dependents_done() (airflow.models.TaskInstance method) + + B - - - bag_dag() (airflow.models.DagBag method) - - - - BaseOperator (class in airflow.models), [1] - - - - BaseSensorOperator (class in airflow.operators.sensors) - - - - BashOperator (class in airflow.operators) - - - - BigQueryHook (class in airflow.contrib.hooks) - - - - - - BigQueryOperator (class in airflow.contrib.operators.bigquery_operator) - - - - BigQueryToCloudStorageOperator (class in airflow.contrib.operators.bigquery_to_gcs) - - - - BranchPythonOperator (class in airflow.operators) - - - - bulk_dump() (airflow.hooks.DbApiHook method) - - - - bulk_load() (airflow.hooks.DbApiHook method) - - - - - (airflow.hooks.MySqlHook method) - - - - + + bag_dag() (airflow.models.DagBag method) + + BaseOperator (class in airflow.models), [1] + + BaseSensorOperator (class in airflow.operators.sensors) + + BashOperator (class in airflow.operators) + + + + BranchPythonOperator (class in airflow.operators) + + bulk_dump() (airflow.hooks.DbApiHook method) + + bulk_load() (airflow.hooks.DbApiHook method) + + +(airflow.hooks.MySqlHook method) + + + C - - - CeleryExecutor (class in airflow.executors) - - - - check_for_bucket() (airflow.hooks.S3Hook method) - - - - check_for_key() (airflow.hooks.S3Hook method) - - - - check_for_named_partition() (airflow.hooks.HiveMetastoreHook method) - - - - check_for_partition() (airflow.hooks.HiveMetastoreHook method) - - - - check_for_path() (airflow.hooks.WebHDFSHook method) - - - - check_for_prefix() (airflow.hooks.S3Hook method) - - - - check_for_wildcard_key() (airflow.hooks.S3Hook method) - - - - check_output() (airflow.contrib.hooks.SSHHook method) - - - - clear() (airflow.models.BaseOperator method) - - - - - (airflow.models.DAG method) - - - - - clear_xcom_data() (airflow.models.TaskInstance method) - - - - cli() (airflow.models.DAG method) - - - - close_conn() (airflow.contrib.hooks.FTPHook method) - - - - - - closest_ds_partition() (in module airflow.macros.hive) - - - - CloudantHook (class in airflow.contrib.hooks) - - - - collect_dags() (airflow.models.DagBag method) - - - - command() (airflow.models.TaskInstance method) - - - - concurrency_reached (airflow.models.DAG attribute) - - - - Connection (class in airflow.models) - - - - construct_api_call_params() (airflow.operators.SlackAPIOperator method) - - - - construct_ingest_query() (airflow.hooks.DruidHook method) - - - -
[07/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/js/theme.js -- diff --git a/_static/js/theme.js b/_static/js/theme.js index 48a9f06..af661a9 100644 --- a/_static/js/theme.js +++ b/_static/js/theme.js @@ -13,33 +13,36 @@ function ThemeNav () { winPosition: 0, winHeight: null, docHeight: null, -isRunning: null +isRunning: false }; nav.enable = function () { var self = this; -jQuery(function ($) { -self.init($); - -self.reset(); -self.win.on('hashchange', self.reset); - -// Set scroll monitor -self.win.on('scroll', function () { -if (!self.linkScroll) { -self.winScroll = true; -} -}); -setInterval(function () { if (self.winScroll) self.onScroll(); }, 25); - -// Set resize monitor -self.win.on('resize', function () { -self.winResize = true; +if (!self.isRunning) { +self.isRunning = true; +jQuery(function ($) { +self.init($); + +self.reset(); +self.win.on('hashchange', self.reset); + +// Set scroll monitor +self.win.on('scroll', function () { +if (!self.linkScroll) { +self.winScroll = true; +} +}); +setInterval(function () { if (self.winScroll) self.onScroll(); }, 25); + +// Set resize monitor +self.win.on('resize', function () { +self.winResize = true; +}); +setInterval(function () { if (self.winResize) self.onResize(); }, 25); +self.onResize(); }); -setInterval(function () { if (self.winResize) self.onResize(); }, 25); -self.onResize(); -}); +}; }; nav.init = function ($) { @@ -95,6 +98,19 @@ function ThemeNav () { try { var link = $('.wy-menu-vertical') .find('[href="' + anchor + '"]'); +// If we didn't find a link, it may be because we clicked on +// something that is not in the sidebar (eg: when using +// sphinxcontrib.httpdomain it generates headerlinks but those +// aren't picked up and placed in the toctree). So let's find +// the closest header in the document and try with that one. +if (link.length === 0) { + var doc_link = $('.document a[href="' + anchor + '"]'); + var closest_section = doc_link.closest('div.section'); + // Try again with the closest section entry. + link = $('.wy-menu-vertical') +.find('[href="#' + closest_section.attr("id") + '"]'); + +} $('.wy-menu-vertical li.toctree-l1 li.current') .removeClass('current'); link.closest('li.toctree-l2').addClass('current'); http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/minus.png -- diff --git a/_static/minus.png b/_static/minus.png index 0f22b16..d96755f 100644 Binary files a/_static/minus.png and b/_static/minus.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/plus.png -- diff --git a/_static/plus.png b/_static/plus.png index 0cfe084..7107cec 100644 Binary files a/_static/plus.png and b/_static/plus.png differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_static/searchtools.js -- diff --git a/_static/searchtools.js b/_static/searchtools.js index 066857c..bbfb3ac 100644 --- a/_static/searchtools.js +++ b/_static/searchtools.js @@ -226,6 +226,106 @@ var Scorer = { }; + + + +var splitChars = (function() { +var result = {}; +var singles = [96, 180, 187, 191, 215, 247, 749, 885, 903, 907, 909, 930, 1014, 1648, + 1748, 1809, 2416, 2473, 2481, 2526, 2601, 2609, 2612, 2615, 2653, 2702, + 2706, 2729, 2737, 2740, 2857, 2865, 2868, 2910, 2928, 2948, 2961, 2971, + 2973, 3085, 3089, 3113, 3124, 3213, 3217, 3241, 3252, 3295, 3341, 3345, + 3369, 3506, 3516, 3633, 3715, 3721, 3736, 3744, 3748, 3750, 3756, 3761, + 3781, 3912, 4239, 4347, 4681, 4695, 4697, 4745, 4785, 4799, 4801, 4823, + 4881, 5760, 5901, 5997, 6313, 7405, 8024, 8026, 8028, 8030, 8117, 8125, + 8133, 8181, 8468, 8485, 8487, 8489, 8494, 8527, 11311, 11359, 11687, 11695, + 11703, 117
[06/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/code.html -- diff --git a/code.html b/code.html index 91192d3..732db2e 100644 --- a/code.html +++ b/code.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,13 +94,15 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference Operators BaseOperator BaseSensorOperator -Operator API -Community-contributed Operators +Operator API +Community-contributed Operators Macros @@ -105,8 +111,8 @@ Models -Hooks -Community contributed hooks +Hooks +Community contributed hooks Executors @@ -127,8 +133,10 @@ - -Airflow + + + Airflow + @@ -137,23 +145,40 @@ - + + + + + + + + + + + + -Docs » - -API Reference + + Docs » + + API Reference + + - - View page source + + View page source + + + http://schema.org/Article";> @@ -187,12 +212,12 @@ to understand the primitive features that can be leveraged in your DAGs. -class airflow.models.BaseOperator(task_id, owner='airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, *args, **kwargs< /em>)[source]¶ +class airflow.models.BaseOperator(task_id, owner='Airflow', email=None, email_on_retry=True, email_on_failure=True, retries=0, retry_delay=datetime.timedelta(0, 300), retry_exponential_backoff=False, max_retry_delay=None, start_date=None, end_date=None, schedule_interval=None, depends_on_past=False, wait_for_downstream=False, dag=None, params=None, default_args=None, adhoc=False, priority_weight=1, queue='default', pool=None, sla=None, execution_timeout=None, on_failure_callback=None, on_success_callback=None, on_retry_callback=None, trigger_rule=u'all_success', resources=None, run_as_user=None, *args, **kwargs)[source]¶ Abstract base class for all operators. Since operators create objects that become node in the dag, BaseOperator contains many recursive methods for dag crawling behavior. To derive this class, you are expected to override the constructor as well as the ‘execute’ method. -Operators derived from this task should perform or trigger certain tasks +Operators derived from this class should perform or trigger certain tasks synchronously (wait for completion). Example of operators could be an operator the runs a Pig job (PigOperator), a sensor operator that waits for a partition to land in Hive (HiveSensorOperator), or one that @@ -261,7 +286,7 @@ way to limit concurrency for certain tasks this represents the timedelta after the period is closed. For example if you set an SLA of 1 hour, the scheduler would send dan email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance -has not succeede yet. +has not succeeded yet. The scheduler pays special attention for jobs with an SLA and sends alert emails for sla misses. SLA misses are also recorded in the database @@ -288,6 +313,7 @@ using the constants defined in the static class airflow.utils.TriggerRule resources (dict) – A map of resource parameter names (the argument names of the Resources constructor) to their values. +run_as_user (str) – unix username to impersonate while running the task @@ -326,9 +352,9 @@ between each tries - -Operator API¶ -Importer that dynamically loads a class and module from its parent. This + +Operator API¶ +Importer that dynamically loads a class and module from its parent. This allows Airflow to support from airflow.operators import BashOperator even though BashOperator is actually in airflow.operators.bash_operator. @@ -427,7 +453,7 @@ DAG. -class airflow.operators.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, *args, **kwargs)[source]¶ +class airflow.operators.EmailOperator(to, subject, html_content, files=None, cc=None, bcc=None, mime_subtype='mixed', *args, **kwargs)[source]¶ Bases: airflow.models.BaseOperator Sends an email. @@ -435,13 +461,13 @@ DAG. Parameters: -to (list or string (comma or semicolon delimited)) – list of emails to send the email to +to (list or string (comma or semicolon delimited)) – list of emails to send the email to su
[14/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/http_operator.html -- diff --git a/_modules/http_operator.html b/_modules/http_operator.html index 00496c9..a22b710 100644 --- a/_modules/http_operator.html +++ b/_modules/http_operator.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,19 +126,36 @@ + + + + + + + + + + + -Docs » - + + Docs » + Module code » - -http_operator + + http_operator + + - + + + + http://schema.org/Article";> @@ -153,10 +178,10 @@ import logging -from airflow.exceptions import AirflowException -from airflow.hooks.http_hook import HttpHook -from airflow.models import BaseOperator -from airflow.utils.decorators import apply_defaults +from airflow.exceptions import AirflowException +from airflow.hooks.http_hook import HttpHook +from airflow.models import BaseOperator +from airflow.utils.decorators import apply_defaults [docs]class SimpleHttpOperator(BaseOperator): @@ -192,11 +217,16 @@ def __init__(self, endpoint, method='POST', - data=None, - headers=None, - response_check=None, - extra_options=None, + data=None, + headers=None, + response_check=None, + extra_options=None, + xcom_push=False, http_conn_id='http_default', *args, **kwargs): +""" +If xcom_push is True, response of an HTTP request will also +be pushed to an XCom. +""" super(SimpleHttpOperator, self).__init__(*args, **kwargs) self.http_conn_id = http_conn_id self.method = method @@ -205,6 +235,7 @@ self.data = data or {} self.response_check = response_check self.extra_options = extra_options or {} +self.xcom_push_flag = xcom_push def execute(self, context): http = HttpHook(self.method, http_conn_id=self.http_conn_id) @@ -215,10 +246,15 @@ self.extra_options) if self.response_check: if not self.response_check(response): -raise AirflowException("Response check returned False.") +raise AirflowException("Response check returned False.") +if self.xcom_push_flag: +return response.text + + + @@ -251,7 +287,8 @@ VERSION:'', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', -HAS_SOURCE: true +HAS_SOURCE: true, +SOURCELINK_SUFFIX: '.txt' }; http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/index.html -- diff --git a/_modules/index.html b/_modules/index.html index d40ab64..b1faeb5 100644 --- a/_modules/index.html +++ b/_modules/index.html @@ -30,6 +30,9 @@ + + @@ -39,6 +42,7 @@ + @@ -89,6 +93,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -103,8 +109,10 @@ - -Airflow + + + Airflow + @@ -117,81 +125,72 @@ + + + + + + + + + + + -Docs » - -Overview: module code + + Docs » + + Overview: module code + + - + + + + http://schema.org/Article";> All modules for which code is available -S3_hook -airflow.contrib.executors.mesos_executor -airflow.contrib.hooks.gcs_hook -airflow.contrib.operators.bigquery_operator -airflow.contrib.operators.bigquery_to_gcs -airflow.contrib.operators.gcs_download_operator +airflow.contrib.executors.mesos_executor airflow.contrib.operators.hipchat_operator -airflow.executors.celery_executor airflow.executors.local_executor airflow.executors.sequential_executor airflow.macros airflow.macros.hive airflow.models -airflow.operators.docker_operator airflow.operators.sensors bash_operator -bigquery_hook -cloudant_hook dagrun_operator dbapi_hook -druid_hook dummy_operator email_operator ftp_hook -gcs_hook generic_transfer -hive_hooks -hive_operator -hive_to_druid -hive_to
[17/22] incubator-airflow-site git commit: Latest docs version as of 1.8.x
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/_modules/airflow/models.html -- diff --git a/_modules/airflow/models.html b/_modules/airflow/models.html index 2f044a4..a94686b 100644 --- a/_modules/airflow/models.html +++ b/_modules/airflow/models.html @@ -30,6 +30,9 @@ + + @@ -40,6 +43,7 @@ + @@ -90,6 +94,8 @@ Scheduling & Triggers Plugins Security +Experimental Rest API +Integration FAQ API Reference @@ -104,8 +110,10 @@ - -Airflow + + + Airflow + @@ -118,19 +126,36 @@ + + + + + + + + + + + -Docs » - + + Docs » + Module code » - -airflow.models + + airflow.models + + - + + + + http://schema.org/Article";> @@ -150,24 +175,25 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -from __future__ import absolute_import -from __future__ import division -from __future__ import print_function -from __future__ import unicode_literals +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals -from future.standard_library import install_aliases +from future.standard_library import install_aliases install_aliases() -from builtins import str -from builtins import object, bytes +from builtins import str +from builtins import object, bytes import copy -from collections import namedtuple -from datetime import datetime, timedelta +from collections import namedtuple +from datetime import datetime, timedelta import dill import functools import getpass import imp import importlib +import inspect import zipfile import jinja2 import json @@ -181,66 +207,70 @@ import textwrap import traceback import warnings -from urllib.parse import urlparse +import hashlib -from sqlalchemy import ( +from urllib.parse import urlparse + +from sqlalchemy import ( Column, Integer, String, DateTime, Text, Boolean, ForeignKey, PickleType, Index, Float) -from sqlalchemy import case, func, or_, and_ -from sqlalchemy.ext.declarative import declarative_base, declared_attr -from sqlalchemy.dialects.mysql import LONGTEXT -from sqlalchemy.orm import relationship, synonym +from sqlalchemy import func, or_, and_ +from sqlalchemy.ext.declarative import declarative_base, declared_attr +from sqlalchemy.dialects.mysql import LONGTEXT +from sqlalchemy.orm import reconstructor, relationship, synonym -from croniter import croniter +from croniter import croniter import six -from airflow import settings, utils -from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor -from airflow import configuration -from airflow.exceptions import AirflowException, AirflowSkipException -from airflow.dag.base_dag import BaseDag, BaseDagBag -from airflow.utils.dates import cron_presets, date_range as utils_date_range -from airflow.utils.db import provide_session -from airflow.utils.decorators import apply_defaults -from airflow.utils.email import send_email -from airflow.utils.helpers import ( +from airflow import settings, utils +from airflow.executors import DEFAULT_EXECUTOR, LocalExecutor +from airflow import configuration +from airflow.exceptions import AirflowException, AirflowSkipException, AirflowTaskTimeout +from airflow.dag.base_dag import BaseDag, BaseDagBag +from airflow.ti_deps.deps.not_in_retry_period_dep import NotInRetryPeriodDep +from airflow.ti_deps.deps.prev_dagrun_dep import PrevDagrunDep +from airflow.ti_deps.deps.trigger_rule_dep import TriggerRuleDep +from airflow.ti_deps.dep_context import DepContext, QUEUE_DEPS, RUN_DEPS +from airflow.utils.dates import cron_presets, date_range as utils_date_range +from airflow.utils.db import provide_session +from airflow.utils.decorators import apply_defaults +from airflow.utils.email import send_email +from airflow.utils.helpers import ( as_tuple, is_container, is_in, validate_key, pprinttable) -from airflow.utils.logging import LoggingMixin -from airflow.utils.operator_resources import Resources -from airflow.utils.state import State -from airflow.utils.timeout import timeout -from airflow.utils.trigger_rule import TriggerRule +from airflow.utils.logging import LoggingMixin +from airflow.utils.operator_resources import Resources +from airflow.utils.state import State +from airflow.utils.timeout import timeout +from airflow.utils.trigger_rule import TriggerRule Base = declarative_base() ID_LEN = 250 -SQL_ALCHEMY_CONN = configuration.get('core', 'SQL_ALCHEMY_CONN') -DAGS_FOLDER = os.path.expanduser(configuration.get
[jira] [Created] (AIRFLOW-944) Docker operator does not work with Docker >= 1.19
Ludovic Claude created AIRFLOW-944: -- Summary: Docker operator does not work with Docker >= 1.19 Key: AIRFLOW-944 URL: https://issues.apache.org/jira/browse/AIRFLOW-944 Project: Apache Airflow Issue Type: Bug Components: docker Affects Versions: Airflow 1.7.1.3, 1.8.0rc4 Environment: Ubuntu 16.04 Reporter: Ludovic Claude Docker operator does not work when mem_limit is set and Docker version 1.19 or more recent is used. Here are the logs, I have seen this issue with Airflow 1.7.1.3 and Airflow 1.8.0 rc4. [2017-03-06 11:37:54,895] {base_task_runner.py:95} INFO - Subtask: [2017-03-06 11:37:54,895] {docker_operator.py:132} INFO - Starting docker container from image hbpmip/mipmap [2017-03-06 11:37:54,903] {base_task_runner.py:95} INFO - Subtask: [2017-03-06 11:37:54,902] {models.py:1417} ERROR - mem_limit has been moved to host_config in API version 1.19 [2017-03-06 11:37:54,903] {base_task_runner.py:95} INFO - Subtask: Traceback (most recent call last): [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1369, in run [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: result = task_copy.execute(context=context) [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: File "/tmp/src/airflow-imaging-plugins/airflow_pipeline/operators/docker_pipeline_operator.py", line 191, in execute [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: logs = super(DockerPipelineOperator, self).execute(context) [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python3.5/dist-packages/airflow/operators/docker_operator.py", line 172, in execute [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: user=self.user [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python3.5/dist-packages/docker/api/container.py", line 133, in create_container [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: volume_driver, stop_signal, networking_config, [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python3.5/dist-packages/docker/api/container.py", line 138, in create_container_config [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: return utils.create_container_config(self._version, *args, **kwargs) [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: File "/usr/local/lib/python3.5/dist-packages/docker/utils/utils.py", line 1041, in create_container_config [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: 'mem_limit has been moved to host_config in API version 1.19' [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: docker.errors.InvalidVersion: mem_limit has been moved to host_config in API version 1.19 [2017-03-06 11:37:54,904] {base_task_runner.py:95} INFO - Subtask: [2017-03-06 11:37:54,903] {models.py:1433} INFO - Marking task as UP_FOR_RETRY [2017-03-06 11:37:54,912] {base_task_runner.py:95} INFO - Subtask: [2017-03-06 11:37:54,912] {models.py:1462} ERROR - mem_limit has been moved to host_config in API version 1.19 -- This message was sent by Atlassian JIRA (v6.3.15#6346)