http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/concepts.html ---------------------------------------------------------------------- diff --git a/concepts.html b/concepts.html index 15b2b4d..1ed51a5 100644 --- a/concepts.html +++ b/concepts.html @@ -30,6 +30,9 @@ + <link rel="index" title="Index" + href="genindex.html"/> + <link rel="search" title="Search" href="search.html"/> <link rel="top" title="Airflow Documentation" href="index.html"/> <link rel="next" title="Data Profiling" href="profiling.html"/> <link rel="prev" title="UI / Screenshots" href="ui.html"/> @@ -41,6 +44,7 @@ <body class="wy-body-for-nav" role="document"> + <div class="wy-grid-for-nav"> @@ -114,6 +118,7 @@ <li class="toctree-l3"><a class="reference internal" href="#subdags">SubDAGs</a></li> <li class="toctree-l3"><a class="reference internal" href="#slas">SLAs</a></li> <li class="toctree-l3"><a class="reference internal" href="#trigger-rules">Trigger Rules</a></li> +<li class="toctree-l3"><a class="reference internal" href="#latest-run-only">Latest Run Only</a></li> <li class="toctree-l3"><a class="reference internal" href="#zombies-undeads">Zombies & Undeads</a></li> <li class="toctree-l3"><a class="reference internal" href="#cluster-policy">Cluster Policy</a></li> <li class="toctree-l3"><a class="reference internal" href="#documentation-notes">Documentation & Notes</a></li> @@ -128,6 +133,8 @@ <li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li> <li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li> <li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li> +<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li> +<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li> <li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li> <li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li> </ul> @@ -142,8 +149,10 @@ <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="index.html">Airflow</a> + + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="index.html">Airflow</a> + </nav> @@ -152,23 +161,40 @@ <div class="rst-content"> - + + + + + + + + + + + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> - <li><a href="index.html">Docs</a> »</li> - - <li>Concepts</li> + + <li><a href="index.html">Docs</a> »</li> + + <li>Concepts</li> + + <li class="wy-breadcrumbs-aside"> - - <a href="_sources/concepts.txt" rel="nofollow"> View page source</a> + + <a href="_sources/concepts.rst.txt" rel="nofollow"> View page source</a> </li> + </ul> + + <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> @@ -341,8 +367,7 @@ object is always returned. For example:</p> <p>We can put this all together to build a simple pipeline:</p> <div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="k">with</span> <span class="n">DAG</span><span class="p">(</span><span class="s1">'my_dag'</span><span class="p">,</span> <span class="n">start_date</span><span class="o">=</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">))</span> <span class="k">as</span> <span class="n">dag</span><span class="p">:</span> <span class="p">(</span> - <span class="n">dag</span> - <span class="o">>></span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span> + <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'dummy_1'</span><span class="p">)</span> <span class="o">>></span> <span class="n">BashOperator</span><span class="p">(</span> <span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_1'</span><span class="p">,</span> <span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "HELLO!"'</span><span class="p">)</span> @@ -452,10 +477,11 @@ variables from the operating system. The environment variable needs to be prefixed with <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> to be considered a connection. When referencing the connection in the Airflow pipeline, the <code class="docutils literal"><span class="pre">conn_id</span></code> should be the name of the variable without the prefix. For example, if the <code class="docutils literal"><span class="pre">conn_id</span></code> -is named <code class="docutils literal"><span class="pre">POSTGRES_MASTER</span></code> the environment variable should be named -<code class="docutils literal"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code>. Airflow assumes the value returned -from the environment variable to be in a URI format -(e.g. <code class="docutils literal"><span class="pre">postgres://user:password@localhost:5432/master</span></code>).</p> +is named <code class="docutils literal"><span class="pre">postgres_master</span></code> the environment variable should be named +<code class="docutils literal"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code> (note that the environment variable must be +all uppercase). Airflow assumes the value returned from the environment +variable to be in a URI format (e.g. +<code class="docutils literal"><span class="pre">postgres://user:password@localhost:5432/master</span></code> or <code class="docutils literal"><span class="pre">s3://accesskey:secretkey@S3</span></code>).</p> </div> <div class="section" id="queues"> <h3>Queues<a class="headerlink" href="#queues" title="Permalink to this headline">¶</a></h3> @@ -674,6 +700,71 @@ while creating tasks:</p> that, when set to <code class="docutils literal"><span class="pre">True</span></code>, keeps a task from getting triggered if the previous schedule for the task hasn’t succeeded.</p> </div> +<div class="section" id="latest-run-only"> +<h3>Latest Run Only<a class="headerlink" href="#latest-run-only" title="Permalink to this headline">¶</a></h3> +<p>Standard workflow behavior involves running a series of tasks for a +particular date/time range. Some workflows, however, perform tasks that +are independent of run time but need to be run on a schedule, much like a +standard cron job. In these cases, backfills or running jobs missed during +a pause just wastes CPU cycles.</p> +<p>For situations like this, you can use the <code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> to skip +tasks that are not being run during the most recent scheduled run for a +DAG. The <code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> skips all immediate downstream tasks, and +itself, if the time right now is not between its <code class="docutils literal"><span class="pre">execution_time</span></code> and the +next scheduled <code class="docutils literal"><span class="pre">execution_time</span></code>.</p> +<p>One must be aware of the interaction between skipped tasks and trigger +rules. Skipped tasks will cascade through trigger rules <code class="docutils literal"><span class="pre">all_success</span></code> +and <code class="docutils literal"><span class="pre">all_failed</span></code> but not <code class="docutils literal"><span class="pre">all_done</span></code>, <code class="docutils literal"><span class="pre">one_failed</span></code>, <code class="docutils literal"><span class="pre">one_success</span></code>, +and <code class="docutils literal"><span class="pre">dummy</span></code>. If you would like to use the <code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> with +trigger rules that do not cascade skips, you will need to ensure that the +<code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code> is <strong>directly</strong> upstream of the task you would like +to skip.</p> +<p>It is possible, through use of trigger rules to mix tasks that should run +in the typical date/time dependent mode and those using the +<code class="docutils literal"><span class="pre">LatestOnlyOperator</span></code>.</p> +<p>For example, consider the following dag:</p> +<div class="code python highlight-default"><div class="highlight"><pre><span></span><span class="c1">#dags/latest_only_with_trigger.py</span> +<span class="kn">import</span> <span class="nn">datetime</span> <span class="k">as</span> <span class="nn">dt</span> + +<span class="kn">from</span> <span class="nn">airflow.models</span> <span class="k">import</span> <span class="n">DAG</span> +<span class="kn">from</span> <span class="nn">airflow.operators.dummy_operator</span> <span class="k">import</span> <span class="n">DummyOperator</span> +<span class="kn">from</span> <span class="nn">airflow.operators.latest_only_operator</span> <span class="k">import</span> <span class="n">LatestOnlyOperator</span> +<span class="kn">from</span> <span class="nn">airflow.utils.trigger_rule</span> <span class="k">import</span> <span class="n">TriggerRule</span> + + +<span class="n">dag</span> <span class="o">=</span> <span class="n">DAG</span><span class="p">(</span> + <span class="n">dag_id</span><span class="o">=</span><span class="s1">'latest_only_with_trigger'</span><span class="p">,</span> + <span class="n">schedule_interval</span><span class="o">=</span><span class="n">dt</span><span class="o">.</span><span class="n">timedelta</span><span class="p">(</span><span class="n">hours</span><span class="o">=</span><span class="mi">4</span><span class="p">),</span> + <span class="n">start_date</span><span class="o">=</span><span class="n">dt</span><span class="o">.</span><span class="n">datetime</span><span class="p">(</span><span class="mi">2016</span><span class="p">,</span> <span class="mi">9</span><span class="p">,</span> <span class="mi">20</span><span class="p">),</span> +<span class="p">)</span> + +<span class="n">latest_only</span> <span class="o">=</span> <span class="n">LatestOnlyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'latest_only'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> + +<span class="n">task1</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task1'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +<span class="n">task1</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">latest_only</span><span class="p">)</span> + +<span class="n">task2</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task2'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> + +<span class="n">task3</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task3'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +<span class="n">task3</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">([</span><span class="n">task1</span><span class="p">,</span> <span class="n">task2</span><span class="p">])</span> + +<span class="n">task4</span> <span class="o">=</span> <span class="n">DummyOperator</span><span class="p">(</span><span class="n">task_id</span><span class="o">=</span><span class="s1">'task4'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">,</span> + <span class="n">trigger_rule</span><span class="o">=</span><span class="n">TriggerRule</span><span class="o">.</span><span class="n">ALL_DONE</span><span class="p">)</span> +<span class="n">task4</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">([</span><span class="n">task1</span><span class="p">,</span> <span class="n">task2</span><span class="p">])</span> +</pre></div> +</div> +<p>In the case of this dag, the <code class="docutils literal"><span class="pre">latest_only</span></code> task will show up as skipped +for all runs except the latest run. <code class="docutils literal"><span class="pre">task1</span></code> is directly downstream of +<code class="docutils literal"><span class="pre">latest_only</span></code> and will also skip for all runs except the latest. +<code class="docutils literal"><span class="pre">task2</span></code> is entirely independent of <code class="docutils literal"><span class="pre">latest_only</span></code> and will run in all +scheduled periods. <code class="docutils literal"><span class="pre">task3</span></code> is downstream of <code class="docutils literal"><span class="pre">task1</span></code> and <code class="docutils literal"><span class="pre">task2</span></code> and +because of the default <code class="docutils literal"><span class="pre">trigger_rule</span></code> being <code class="docutils literal"><span class="pre">all_success</span></code> will receive +a cascaded skip from <code class="docutils literal"><span class="pre">task1</span></code>. <code class="docutils literal"><span class="pre">task4</span></code> is downstream of <code class="docutils literal"><span class="pre">task1</span></code> and +<code class="docutils literal"><span class="pre">task2</span></code> but since its <code class="docutils literal"><span class="pre">trigger_rule</span></code> is set to <code class="docutils literal"><span class="pre">all_done</span></code> it will +trigger as soon as <code class="docutils literal"><span class="pre">task1</span></code> has been skipped (a valid completion state) +and <code class="docutils literal"><span class="pre">task2</span></code> has succeeded.</p> +<img alt="_images/latest_only_with_trigger.png" src="_images/latest_only_with_trigger.png" /> +</div> <div class="section" id="zombies-undeads"> <h3>Zombies & Undeads<a class="headerlink" href="#zombies-undeads" title="Permalink to this headline">¶</a></h3> <p>Task instances die all the time, usually as part of their normal life cycle, @@ -842,15 +933,18 @@ pure python modules can be packaged.</p> </div> + <div class="articleComments"> + + </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - <a href="profiling.html" class="btn btn-neutral float-right" title="Data Profiling" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + <a href="profiling.html" class="btn btn-neutral float-right" title="Data Profiling" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> - <a href="ui.html" class="btn btn-neutral" title="UI / Screenshots" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + <a href="ui.html" class="btn btn-neutral" title="UI / Screenshots" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> </div> @@ -883,7 +977,8 @@ pure python modules can be packaged.</p> VERSION:'', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', - HAS_SOURCE: true + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' }; </script> <script type="text/javascript" src="_static/jquery.js"></script>
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/configuration.html ---------------------------------------------------------------------- diff --git a/configuration.html b/configuration.html index ff5bd46..dc4577c 100644 --- a/configuration.html +++ b/configuration.html @@ -30,6 +30,9 @@ + <link rel="index" title="Index" + href="genindex.html"/> + <link rel="search" title="Search" href="search.html"/> <link rel="top" title="Airflow Documentation" href="index.html"/> <link rel="next" title="UI / Screenshots" href="ui.html"/> <link rel="prev" title="Tutorial" href="tutorial.html"/> @@ -41,6 +44,7 @@ <body class="wy-body-for-nav" role="document"> + <div class="wy-grid-for-nav"> @@ -88,10 +92,12 @@ <li class="toctree-l2"><a class="reference internal" href="#setting-up-a-backend">Setting up a Backend</a></li> <li class="toctree-l2"><a class="reference internal" href="#connections">Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-celery">Scaling Out with Celery</a></li> +<li class="toctree-l2"><a class="reference internal" href="#scaling-out-with-dask">Scaling Out with Dask</a></li> <li class="toctree-l2"><a class="reference internal" href="#logs">Logs</a></li> <li class="toctree-l2"><a class="reference internal" href="#scaling-out-on-mesos-community-contributed">Scaling Out on Mesos (community contributed)</a></li> <li class="toctree-l2"><a class="reference internal" href="#integration-with-systemd">Integration with systemd</a></li> <li class="toctree-l2"><a class="reference internal" href="#integration-with-upstart">Integration with upstart</a></li> +<li class="toctree-l2"><a class="reference internal" href="#test-mode">Test Mode</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="ui.html">UI / Screenshots</a></li> @@ -101,6 +107,8 @@ <li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li> <li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li> <li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li> +<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li> +<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li> <li class="toctree-l1"><a class="reference internal" href="faq.html">FAQ</a></li> <li class="toctree-l1"><a class="reference internal" href="code.html">API Reference</a></li> </ul> @@ -115,8 +123,10 @@ <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="index.html">Airflow</a> + + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="index.html">Airflow</a> + </nav> @@ -125,23 +135,40 @@ <div class="rst-content"> - + + + + + + + + + + + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> - <li><a href="index.html">Docs</a> »</li> - - <li>Configuration</li> + + <li><a href="index.html">Docs</a> »</li> + + <li>Configuration</li> + + <li class="wy-breadcrumbs-aside"> - - <a href="_sources/configuration.txt" rel="nofollow"> View page source</a> + + <a href="_sources/configuration.rst.txt" rel="nofollow"> View page source</a> </li> + </ul> + + <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> @@ -152,7 +179,7 @@ <p>Setting up the sandbox in the <a class="reference internal" href="start.html"><span class="doc">Quick Start</span></a> section was easy; building a production-grade environment requires a bit more work!</p> <div class="section" id="setting-configuration-options"> -<h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">¶</a></h2> +<span id="setting-options"></span><h2>Setting Configuration Options<a class="headerlink" href="#setting-configuration-options" title="Permalink to this headline">¶</a></h2> <p>The first time you run Airflow, it will create a file called <code class="docutils literal"><span class="pre">airflow.cfg</span></code> in your <code class="docutils literal"><span class="pre">$AIRFLOW_HOME</span></code> directory (<code class="docutils literal"><span class="pre">~/airflow</span></code> by default). This file contains Airflow’s configuration and you can edit it to change any of the settings. You can also set options with environment variables by using this format: @@ -258,6 +285,33 @@ its direction.</p> to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code> to start a Flower web server.</p> </div> +<div class="section" id="scaling-out-with-dask"> +<h2>Scaling Out with Dask<a class="headerlink" href="#scaling-out-with-dask" title="Permalink to this headline">¶</a></h2> +<p><code class="docutils literal"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p> +<p>Dask clusters can be run on a single machine or on remote networks. For complete +details, consult the <a class="reference external" href="https://distributed.readthedocs.io/">Distributed documentation</a>.</p> +<p>To create a cluster, first start a Scheduler:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span> +<span class="nv">DASK_HOST</span><span class="o">=</span>127.0.0.1 +<span class="nv">DASK_PORT</span><span class="o">=</span>8786 + +dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv">$DASK_PORT</span> +</pre></div> +</div> +<p>Next start at least one Worker on any machine that can connect to the host:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span> +</pre></div> +</div> +<p>Edit your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal"><span class="pre">DaskExecutor</span></code> and provide +the Dask Scheduler address in the <code class="docutils literal"><span class="pre">[dask]</span></code> section.</p> +<p>Please note:</p> +<ul class="simple"> +<li>Each Dask worker must be able to import Airflow and any dependencies you +require.</li> +<li>Dask does not support queues. If an Airflow task was created with a queue, a +warning will be raised but the task will be submitted to the cluster.</li> +</ul> +</div> <div class="section" id="logs"> <h2>Logs<a class="headerlink" href="#logs" title="Permalink to this headline">¶</a></h2> <p>Users can specify a logs folder in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>. By default, it is in @@ -346,19 +400,37 @@ integrated with upstart</p> </pre></div> </div> </div> +<div class="section" id="test-mode"> +<h2>Test Mode<a class="headerlink" href="#test-mode" title="Permalink to this headline">¶</a></h2> +<p>Airflow has a fixed set of “test mode” configuration options. You can load these +at any time by calling <code class="docutils literal"><span class="pre">airflow.configuration.load_test_config()</span></code> (note this +operation is not reversible!). However, some options (like the DAG_FOLDER) are +loaded before you have a chance to call load_test_config(). In order to eagerly load +the test configuration, set test_mode in airflow.cfg:</p> +<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="o">[</span>tests<span class="o">]</span> +<span class="nv">unit_test_mode</span> <span class="o">=</span> True +</pre></div> +</div> +<p>Due to Airflow’s automatic environment variable expansion (see <a class="reference internal" href="#setting-options"><span class="std std-ref">Setting Configuration Options</span></a>), +you can also set the env var <code class="docutils literal"><span class="pre">AIRFLOW__CORE__UNIT_TEST_MODE</span></code> to temporarily overwrite +airflow.cfg.</p> +</div> </div> </div> + <div class="articleComments"> + + </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - <a href="ui.html" class="btn btn-neutral float-right" title="UI / Screenshots" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + <a href="ui.html" class="btn btn-neutral float-right" title="UI / Screenshots" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> - <a href="tutorial.html" class="btn btn-neutral" title="Tutorial" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + <a href="tutorial.html" class="btn btn-neutral" title="Tutorial" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> </div> @@ -391,7 +463,8 @@ integrated with upstart</p> VERSION:'', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', - HAS_SOURCE: true + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' }; </script> <script type="text/javascript" src="_static/jquery.js"></script> http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/9c75ee9e/faq.html ---------------------------------------------------------------------- diff --git a/faq.html b/faq.html index 623e37f..7b6c8ab 100644 --- a/faq.html +++ b/faq.html @@ -30,9 +30,12 @@ + <link rel="index" title="Index" + href="genindex.html"/> + <link rel="search" title="Search" href="search.html"/> <link rel="top" title="Airflow Documentation" href="index.html"/> <link rel="next" title="API Reference" href="code.html"/> - <link rel="prev" title="Security" href="security.html"/> + <link rel="prev" title="Integration" href="integration.html"/> <script src="_static/js/modernizr.min.js"></script> @@ -41,6 +44,7 @@ <body class="wy-body-for-nav" role="document"> + <div class="wy-grid-for-nav"> @@ -91,6 +95,8 @@ <li class="toctree-l1"><a class="reference internal" href="scheduler.html">Scheduling & Triggers</a></li> <li class="toctree-l1"><a class="reference internal" href="plugins.html">Plugins</a></li> <li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li> +<li class="toctree-l1"><a class="reference internal" href="api.html">Experimental Rest API</a></li> +<li class="toctree-l1"><a class="reference internal" href="integration.html">Integration</a></li> <li class="toctree-l1 current"><a class="current reference internal" href="#">FAQ</a><ul> <li class="toctree-l2"><a class="reference internal" href="#why-isn-t-my-task-getting-scheduled">Why isn’t my task getting scheduled?</a></li> <li class="toctree-l2"><a class="reference internal" href="#how-do-i-trigger-tasks-based-on-another-task-s-failure">How do I trigger tasks based on another task’s failure?</a></li> @@ -113,8 +119,10 @@ <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> - <i data-toggle="wy-nav-top" class="fa fa-bars"></i> - <a href="index.html">Airflow</a> + + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="index.html">Airflow</a> + </nav> @@ -123,23 +131,40 @@ <div class="rst-content"> - + + + + + + + + + + + <div role="navigation" aria-label="breadcrumbs navigation"> + <ul class="wy-breadcrumbs"> - <li><a href="index.html">Docs</a> »</li> - - <li>FAQ</li> + + <li><a href="index.html">Docs</a> »</li> + + <li>FAQ</li> + + <li class="wy-breadcrumbs-aside"> - - <a href="_sources/faq.txt" rel="nofollow"> View page source</a> + + <a href="_sources/faq.rst.txt" rel="nofollow"> View page source</a> </li> + </ul> + + <hr/> </div> <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article"> @@ -161,6 +186,10 @@ may way to confirm that this works both where the scheduler runs as well as where the worker runs.</li> <li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> set properly? The Airflow scheduler triggers the task soon after the <code class="docutils literal"><span class="pre">start_date</span> <span class="pre">+</span> <span class="pre">scheduler_interval</span></code> is passed.</li> +<li>Is your <code class="docutils literal"><span class="pre">schedule_interval</span></code> set properly? The default <code class="docutils literal"><span class="pre">schedule_interval</span></code> +is one day (<code class="docutils literal"><span class="pre">datetime.timedelta(1)</span></code>). You must specify a different <code class="docutils literal"><span class="pre">schedule_interval</span></code> +directly to the DAG object you instantiate, not as a <code class="docutils literal"><span class="pre">default_param</span></code>, as task instances +do not override their parent DAG’s <code class="docutils literal"><span class="pre">schedule_interval</span></code>.</li> <li>Is your <code class="docutils literal"><span class="pre">start_date</span></code> beyond where you can see it in the UI? If you set your it to some time say 3 months ago, you won’t be able to see it in the main view in the UI, but you should be able to see it in the @@ -171,7 +200,7 @@ if you have set <code class="docutils literal"><span class="pre">depends_on_past needs to have succeeded (except if it is the first run for that task). Also, if <code class="docutils literal"><span class="pre">wait_for_downstream=True</span></code>, make sure you understand what it means. -You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Details</span></code> +You can view how these properties are set from the <code class="docutils literal"><span class="pre">Task</span> <span class="pre">Instance</span> <span class="pre">Details</span></code> page for your task.</li> <li>Are the DagRuns you need created and active? A DagRun represents a specific execution of an entire DAG and has a state (running, success, failed, ...). @@ -263,7 +292,7 @@ simple dictionary.</p> command in the queue for it to run remote, on the worker. If using LocalExecutor, that translates into running it in a subprocess pool.</li> <li>Local <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--local</span></code>: starts an <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> -command (described bellow) as a subprocess and is in charge of +command (described below) as a subprocess and is in charge of emitting heartbeats, listening for external kill signals and ensures some cleanup takes place if the subprocess fails</li> <li>Raw <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">run</span> <span class="pre">--raw</span></code> runs the actual operator’s execute method and @@ -274,15 +303,18 @@ performs the actual work</li> </div> + <div class="articleComments"> + + </div> </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - <a href="code.html" class="btn btn-neutral float-right" title="API Reference" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a> + <a href="code.html" class="btn btn-neutral float-right" title="API Reference" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> - <a href="security.html" class="btn btn-neutral" title="Security" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a> + <a href="integration.html" class="btn btn-neutral" title="Integration" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> </div> @@ -315,7 +347,8 @@ performs the actual work</li> VERSION:'', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', - HAS_SOURCE: true + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' }; </script> <script type="text/javascript" src="_static/jquery.js"></script>