http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/kubernetes.html ---------------------------------------------------------------------- diff --git a/kubernetes.html b/kubernetes.html index 9d0ca02..a9bd534 100644 --- a/kubernetes.html +++ b/kubernetes.html @@ -163,9 +163,11 @@ <div class="section" id="kubernetes-executor"> <h1>Kubernetes Executor<a class="headerlink" href="#kubernetes-executor" title="Permalink to this headline">¶</a></h1> <p>The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance.</p> -<p>Example helm charts are available at <cite>scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml</cite> in the source distribution. The volumes are optional and depend on your configuration. There are two volumes available: -- Dags: by storing all the dags onto the persistent disks, all the workers can read the dags from there. Another option is using git-sync, before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod/ -- Logs: by storing the logs onto a persistent disk, all the logs will be available for all the workers and the webserver itself. If you donât configure this, the logs will be lost after the worker pods shuts down. Another option is to use S3/GCS/etc to store the logs.</p> +<p>Example helm charts are available at <cite>scripts/ci/kubernetes/kube/{airflow,volumes,postgres}.yaml</cite> in the source distribution. The volumes are optional and depend on your configuration. There are two volumes available:</p> +<ul class="simple"> +<li>Dags: by storing all the dags onto the persistent disks, all the workers can read the dags from there. Another option is using git-sync, before starting the container, a git pull of the dags repository will be performed and used throughout the lifecycle of the pod.</li> +<li>Logs: by storing the logs onto a persistent disk, all the logs will be available for all the workers and the webserver itself. If you donât configure this, the logs will be lost after the worker pods shuts down. Another option is to use S3/GCS/etc to store the logs.</li> +</ul> </div> <div class="section" id="kubernetes-operator"> <h1>Kubernetes Operator<a class="headerlink" href="#kubernetes-operator" title="Permalink to this headline">¶</a></h1> @@ -237,6 +239,14 @@ <span class="p">}</span> <span class="p">}</span> +<span class="n">tolerations</span> <span class="o">=</span> <span class="p">[</span> + <span class="p">{</span> + <span class="s1">'key'</span><span class="p">:</span> <span class="s2">"key"</span><span class="p">,</span> + <span class="s1">'operator'</span><span class="p">:</span> <span class="s1">'Equal'</span><span class="p">,</span> + <span class="s1">'value'</span><span class="p">:</span> <span class="s1">'value'</span> + <span class="p">}</span> +<span class="p">]</span> + <span class="n">k</span> <span class="o">=</span> <span class="n">KubernetesPodOperator</span><span class="p">(</span><span class="n">namespace</span><span class="o">=</span><span class="s1">'default'</span><span class="p">,</span> <span class="n">image</span><span class="o">=</span><span class="s2">"ubuntu:16.04"</span><span class="p">,</span> <span class="n">cmds</span><span class="o">=</span><span class="p">[</span><span class="s2">"bash"</span><span class="p">,</span> <span class="s2">"-cx"</span><span class="p">],</span> @@ -247,13 +257,16 @@ <span class="n">volume_mounts</span><span class="o">=</span><span class="p">[</span><span class="n">volume_mount</span><span class="p">]</span> <span class="n">name</span><span class="o">=</span><span class="s2">"test"</span><span class="p">,</span> <span class="n">task_id</span><span class="o">=</span><span class="s2">"task"</span><span class="p">,</span> - <span class="n">affinity</span><span class="o">=</span><span class="n">affinity</span> + <span class="n">affinity</span><span class="o">=</span><span class="n">affinity</span><span class="p">,</span> + <span class="n">is_delete_operator_pod</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> + <span class="n">hostnetwork</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> + <span class="n">tolerations</span><span class="o">=</span><span class="n">tolerations</span> <span class="p">)</span> </pre></div> </div> <dl class="class"> <dt id="airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator"> -<em class="property">class </em><code class="descclassname">airflow.contrib.operators.kubernetes_pod_operator.</code><code class="descname">KubernetesPodOperator</code><span class="sig-paren">(</span><em>namespace</em>, <em>image</em>, <em>name</em>, <em>cmds=None</em>, <em>arguments=None</em>, <em>volume_mounts=None</em>, <em>volumes=None</em>, <em>env_vars=None</em>, <em>secrets=None</em>, <em>in_cluster=False</em>, <em>cluster_context=None</em>, <em>labels=None</em>, <em>startup_timeout_seconds=120</em>, <em>get_logs=True</em>, <em>image_pull_policy='IfNotPresent'</em>, <em>annotations=None</em>, <em>resources=None</em>, <em>affinity=None</em>, <em>config_file=None</em>, <em>xcom_push=False</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/kubernetes_pod_operator.html#KubernetesPodOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operator s.kubernetes_pod_operator.KubernetesPodOperator" title="Permalink to this definition">¶</a></dt> +<em class="property">class </em><code class="descclassname">airflow.contrib.operators.kubernetes_pod_operator.</code><code class="descname">KubernetesPodOperator</code><span class="sig-paren">(</span><em>namespace</em>, <em>image</em>, <em>name</em>, <em>cmds=None</em>, <em>arguments=None</em>, <em>volume_mounts=None</em>, <em>volumes=None</em>, <em>env_vars=None</em>, <em>secrets=None</em>, <em>in_cluster=False</em>, <em>cluster_context=None</em>, <em>labels=None</em>, <em>startup_timeout_seconds=120</em>, <em>get_logs=True</em>, <em>image_pull_policy='IfNotPresent'</em>, <em>annotations=None</em>, <em>resources=None</em>, <em>affinity=None</em>, <em>config_file=None</em>, <em>xcom_push=False</em>, <em>node_selectors=None</em>, <em>image_pull_secrets=None</em>, <em>service_account_name='default'</em>, <em>is_delete_operator_pod=False</em>, <em>hostnetwork=False</em>, <em>tolerations=None</em>, <em>*args</em>, <em>**kwargs</em><span class="sig-paren">)</span><a class="reference inte rnal" href="_modules/airflow/contrib/operators/kubernetes_pod_operator.html#KubernetesPodOperator"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator" title="Permalink to this definition">¶</a></dt> <dd><p>Bases: <a class="reference internal" href="code.html#airflow.models.BaseOperator" title="airflow.models.BaseOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">airflow.models.BaseOperator</span></code></a></p> <p>Execute a task in a Kubernetes Pod</p> <table class="docutils field-list" frame="void" rules="none"> @@ -281,10 +294,12 @@ They can be exposed as environment vars or files in a volume.</li> Ignored when in_cluster is True. If None, current-context is used.</li> <li><strong>get_logs</strong> (<em>bool</em>) â get the stdout of the container as logs of the tasks</li> <li><strong>affinity</strong> (<em>dict</em>) â A dict containing a group of affinity scheduling rules</li> +<li><strong>node_selectors</strong> (<em>dict</em>) â A dict containing a group of scheduling rules</li> <li><strong>config_file</strong> (<em>str</em>) â The path to the Kubernetes config file</li> <li><strong>xcom_push</strong> (<em>bool</em>) â If xcom_push is True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.</li> +<li><strong>tolerations</strong> â Kubernetes tolerations</li> </ul> </td> </tr> @@ -296,12 +311,22 @@ XCom when the container completes.</li> </tr> </tbody> </table> +<p>:type list of tolerations</p> +<dl class="method"> +<dt id="airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator.execute"> +<code class="descname">execute</code><span class="sig-paren">(</span><em>context</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/operators/kubernetes_pod_operator.html#KubernetesPodOperator.execute"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.operators.kubernetes_pod_operator.KubernetesPodOperator.execute" title="Permalink to this definition">¶</a></dt> +<dd><p>This is the main method to derive when creating an operator. +Context is the same dictionary used as when rendering jinja templates.</p> +<p>Refer to get_template_context for more context.</p> +</dd></dl> + </dd></dl> <dl class="class"> <dt id="airflow.contrib.kubernetes.secret.Secret"> <em class="property">class </em><code class="descclassname">airflow.contrib.kubernetes.secret.</code><code class="descname">Secret</code><span class="sig-paren">(</span><em>deploy_type</em>, <em>deploy_target</em>, <em>secret</em>, <em>key</em><span class="sig-paren">)</span><a class="reference internal" href="_modules/airflow/contrib/kubernetes/secret.html#Secret"><span class="viewcode-link">[source]</span></a><a class="headerlink" href="#airflow.contrib.kubernetes.secret.Secret" title="Permalink to this definition">¶</a></dt> -<dd><p>Defines Kubernetes Secret Volume</p> +<dd><p>Bases: <code class="xref py py-class docutils literal notranslate"><span class="pre">object</span></code></p> +<p>Defines Kubernetes Secret Volume</p> </dd></dl> </div> @@ -335,20 +360,13 @@ XCom when the container completes.</li> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> +
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/license.html ---------------------------------------------------------------------- diff --git a/license.html b/license.html index 65b90e7..48a5bd1 100644 --- a/license.html +++ b/license.html @@ -381,20 +381,13 @@ - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/lineage.html ---------------------------------------------------------------------- diff --git a/lineage.html b/lineage.html index e58ed3d..201fd2a 100644 --- a/lineage.html +++ b/lineage.html @@ -285,20 +285,13 @@ properly, e.g. in your <cite>airflow.cfg</cite>:</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/objects.inv ---------------------------------------------------------------------- diff --git a/objects.inv b/objects.inv index 1a4c2a2..85f39be 100644 Binary files a/objects.inv and b/objects.inv differ http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/plugins.html ---------------------------------------------------------------------- diff --git a/plugins.html b/plugins.html index bad7f71..2929296 100644 --- a/plugins.html +++ b/plugins.html @@ -95,6 +95,7 @@ <li class="toctree-l2"><a class="reference internal" href="#why-build-on-top-of-airflow">Why build on top of Airflow?</a></li> <li class="toctree-l2"><a class="reference internal" href="#interface">Interface</a></li> <li class="toctree-l2"><a class="reference internal" href="#example">Example</a></li> +<li class="toctree-l2"><a class="reference internal" href="#note-on-role-based-views">Note on role based views</a></li> </ul> </li> <li class="toctree-l1"><a class="reference internal" href="security.html">Security</a></li> @@ -230,26 +231,16 @@ looks like:</p> <span class="c1"># A list of objects created from a class derived</span> <span class="c1"># from flask_admin.BaseView</span> <span class="n">admin_views</span> <span class="o">=</span> <span class="p">[]</span> - <span class="c1"># A list of Blueprint object created from flask.Blueprint</span> + <span class="c1"># A list of Blueprint object created from flask.Blueprint. For use with the flask_admin based GUI</span> <span class="n">flask_blueprints</span> <span class="o">=</span> <span class="p">[]</span> - <span class="c1"># A list of menu links (flask_admin.base.MenuLink)</span> + <span class="c1"># A list of menu links (flask_admin.base.MenuLink). For use with the flask_admin based GUI</span> <span class="n">menu_links</span> <span class="o">=</span> <span class="p">[]</span> + <span class="c1"># A list of dictionaries containing FlaskAppBuilder BaseView object and some metadata. See example below</span> + <span class="n">appbuilder_views</span> <span class="o">=</span> <span class="p">[]</span> + <span class="c1"># A list of dictionaries containing FlaskAppBuilder BaseView object and some metadata. See example below</span> + <span class="n">appbuilder_menu_items</span> <span class="o">=</span> <span class="p">[]</span> </pre></div> </div> -<p>You can derive it by inheritance (please refer to the example below). -Please note <code class="docutils literal notranslate"><span class="pre">name</span></code> inside this class must be specified.</p> -<p>After the plugin is imported into Airflow, -you can invoke it using statement like</p> -<div class="code python highlight-default notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">airflow.</span><span class="p">{</span><span class="nb">type</span><span class="p">,</span> <span class="n">like</span> <span class="s2">"operators"</span><span class="p">,</span> <span class="s2">"sensors"</span><span class="p">}</span><span class="o">.</span><span class="p">{</span><span class="n">name</span> <span class="n">specificed</span> <span class="n">inside</span> <span class="n">the</span> <span class="n">plugin</span> <span class="n">class</span><span class="p">}</span> <span class="kn">import</span> <span class="o">*</span> -</pre></div> -</div> -<p>When you write your own plugins, make sure you understand them well. -There are some essential properties for each type of plugin. -For example,</p> -<ul class="simple"> -<li>For <code class="docutils literal notranslate"><span class="pre">Operator</span></code> plugin, an <code class="docutils literal notranslate"><span class="pre">execute</span></code> method is compulsory.</li> -<li>For <code class="docutils literal notranslate"><span class="pre">Sensor</span></code> plugin, a <code class="docutils literal notranslate"><span class="pre">poke</span></code> method returning a Boolean value is compulsory.</li> -</ul> </div> <div class="section" id="example"> <h2>Example<a class="headerlink" href="#example" title="Permalink to this headline">¶</a></h2> @@ -308,6 +299,22 @@ definitions in Airflow.</p> <span class="n">name</span><span class="o">=</span><span class="s1">'Test Menu Link'</span><span class="p">,</span> <span class="n">url</span><span class="o">=</span><span class="s1">'https://airflow.incubator.apache.org/'</span><span class="p">)</span> +<span class="c1"># Creating a flask appbuilder BaseView</span> +<span class="k">class</span> <span class="nc">TestAppBuilderBaseView</span><span class="p">(</span><span class="n">AppBuilderBaseView</span><span class="p">):</span> + <span class="nd">@expose</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span> + <span class="k">def</span> <span class="nf">test</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span> + <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">render</span><span class="p">(</span><span class="s2">"test_plugin/test.html"</span><span class="p">,</span> <span class="n">content</span><span class="o">=</span><span class="s2">"Hello galaxy!"</span><span class="p">)</span> +<span class="n">v_appbuilder_view</span> <span class="o">=</span> <span class="n">TestAppBuilderBaseView</span><span class="p">()</span> +<span class="n">v_appbuilder_package</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"name"</span><span class="p">:</span> <span class="s2">"Test View"</span><span class="p">,</span> + <span class="s2">"category"</span><span class="p">:</span> <span class="s2">"Test Plugin"</span><span class="p">,</span> + <span class="s2">"view"</span><span class="p">:</span> <span class="n">v_appbuilder_view</span><span class="p">}</span> + +<span class="c1"># Creating a flask appbuilder Menu Item</span> +<span class="n">appbuilder_mitem</span> <span class="o">=</span> <span class="p">{</span><span class="s2">"name"</span><span class="p">:</span> <span class="s2">"Google"</span><span class="p">,</span> + <span class="s2">"category"</span><span class="p">:</span> <span class="s2">"Search"</span><span class="p">,</span> + <span class="s2">"category_icon"</span><span class="p">:</span> <span class="s2">"fa-th"</span><span class="p">,</span> + <span class="s2">"href"</span><span class="p">:</span> <span class="s2">"https://www.google.com"</span><span class="p">}</span> + <span class="c1"># Defining the plugin class</span> <span class="k">class</span> <span class="nc">AirflowTestPlugin</span><span class="p">(</span><span class="n">AirflowPlugin</span><span class="p">):</span> <span class="n">name</span> <span class="o">=</span> <span class="s2">"test_plugin"</span> @@ -319,9 +326,17 @@ definitions in Airflow.</p> <span class="n">admin_views</span> <span class="o">=</span> <span class="p">[</span><span class="n">v</span><span class="p">]</span> <span class="n">flask_blueprints</span> <span class="o">=</span> <span class="p">[</span><span class="n">bp</span><span class="p">]</span> <span class="n">menu_links</span> <span class="o">=</span> <span class="p">[</span><span class="n">ml</span><span class="p">]</span> + <span class="n">appbuilder_views</span> <span class="o">=</span> <span class="p">[</span><span class="n">v_appbuilder_package</span><span class="p">]</span> + <span class="n">appbuilder_menu_items</span> <span class="o">=</span> <span class="p">[</span><span class="n">appbuilder_mitem</span><span class="p">]</span> </pre></div> </div> </div> +<div class="section" id="note-on-role-based-views"> +<h2>Note on role based views<a class="headerlink" href="#note-on-role-based-views" title="Permalink to this headline">¶</a></h2> +<p>Airflow 1.10 introduced role based views using FlaskAppBuilder. You can configure which UI is used by setting +rbac = True. To support plugin views and links for both versions of the UI and maintain backwards compatibility, +the fields appbuilder_views and appbuilder_menu_items were added to the AirflowTestPlugin class.</p> +</div> </div> @@ -362,20 +377,13 @@ definitions in Airflow.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/profiling.html ---------------------------------------------------------------------- diff --git a/profiling.html b/profiling.html index 0c63b23..4748961 100644 --- a/profiling.html +++ b/profiling.html @@ -242,20 +242,13 @@ directly in the URL.</p> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/project.html ---------------------------------------------------------------------- diff --git a/project.html b/project.html index c0da142..8e43fee 100644 --- a/project.html +++ b/project.html @@ -196,7 +196,6 @@ the Airbnb Github and announced in June 2015.</p> <li>@fokko (Fokko Driesprong)</li> <li>@ash (Ash Berlin-Taylor)</li> <li>@kaxilnaik (Kaxil Naik)</li> -<li>@feng-tao (Tao Feng)</li> </ul> <p>For the full list of contributors, take a look at <a class="reference external" href="https://github.com/apache/incubator-airflow/graphs/contributors";>Airflowâs Github Contributor page:</a></p> @@ -258,20 +257,13 @@ to subscribe to each)</li> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/py-modindex.html ---------------------------------------------------------------------- diff --git a/py-modindex.html b/py-modindex.html index 60a9c43..2352151 100644 --- a/py-modindex.html +++ b/py-modindex.html @@ -222,20 +222,13 @@ - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/scheduler.html ---------------------------------------------------------------------- diff --git a/scheduler.html b/scheduler.html index 3fb2968..760a234 100644 --- a/scheduler.html +++ b/scheduler.html @@ -180,7 +180,7 @@ Airflow production environment. To kick it off, all you need to do is execute <code class="docutils literal notranslate"><span class="pre">airflow</span> <span class="pre">scheduler</span></code>. It will use the configuration specified in <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code>.</p> <p>Note that if you run a DAG on a <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> of one day, -the run stamped <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code> will be triggered soon after <code class="docutils literal notranslate"><span class="pre">2016-01-01T23:59</span></code>. +the run stamped <code class="docutils literal notranslate"><span class="pre">2016-01-01</span></code> will be trigger soon after <code class="docutils literal notranslate"><span class="pre">2016-01-01T23:59</span></code>. In other words, the job instance is started once the period it covers has ended.</p> <p><strong>Letâs Repeat That</strong> The scheduler runs your job one <code class="docutils literal notranslate"><span class="pre">schedule_interval</span></code> AFTER the @@ -246,6 +246,8 @@ DAGs</td> </tr> </tbody> </table> +<p><strong>Note</strong>: Use <code class="docutils literal notranslate"><span class="pre">schedule_interval=None</span></code> and not <code class="docutils literal notranslate"><span class="pre">schedule_interval='None'</span></code> when +you donât want to schedule your DAG.</p> <p>Your DAG will be instantiated for each schedule, while creating a <code class="docutils literal notranslate"><span class="pre">DAG</span> <span class="pre">Run</span></code> entry for each schedule.</p> <p>DAG runs have a state associated to them (running, failed, success) and @@ -308,7 +310,6 @@ running an <code class="docutils literal notranslate"><span class="pre">airflow< specific <code class="docutils literal notranslate"><span class="pre">run_id</span></code>. The <code class="docutils literal notranslate"><span class="pre">DAG</span> <span class="pre">Runs</span></code> created externally to the scheduler get associated to the triggerâs timestamp, and will be displayed in the UI alongside scheduled <code class="docutils literal notranslate"><span class="pre">DAG</span> <span class="pre">runs</span></code>.</p> -<p>In addition, you can also manually trigger a <code class="docutils literal notranslate"><span class="pre">DAG</span> <span class="pre">Run</span></code> using the web UI (tab âDAGsâ -> column âLinksâ -> button âTrigger Dagâ).</p> </div> <div class="section" id="to-keep-in-mind"> <h2>To Keep in Mind<a class="headerlink" href="#to-keep-in-mind" title="Permalink to this headline">¶</a></h2> @@ -334,7 +335,6 @@ flags for including upstream and downstream relatives, and targeting task instan states (<code class="docutils literal notranslate"><span class="pre">failed</span></code>, or <code class="docutils literal notranslate"><span class="pre">success</span></code>)</li> <li>Clearing a task instance will no longer delete the task instance record. Instead it updates max_tries and set the current task instance state to be None.</li> -<li>Marking task instances as failed can be done through the UI. This can be used to stop running task instances.</li> <li>Marking task instances as successful can be done through the UI. This is mostly to fix false negatives, or for instance when the fix has been applied outside of Airflow.</li> <li>The <code class="docutils literal notranslate"><span class="pre">airflow</span> <span class="pre">backfill</span></code> CLI subcommand has a flag to <code class="docutils literal notranslate"><span class="pre">--mark_success</span></code> and allows selecting @@ -381,20 +381,13 @@ subsections of the DAG as well as specifying date ranges.</li> - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/1f06fa0e/search.html ---------------------------------------------------------------------- diff --git a/search.html b/search.html index bbeec88..561f8c7 100644 --- a/search.html +++ b/search.html @@ -200,21 +200,14 @@ - <script type="text/javascript"> - var DOCUMENTATION_OPTIONS = { - URL_ROOT:'./', - VERSION:'', - LANGUAGE:'None', - COLLAPSE_INDEX:false, - FILE_SUFFIX:'.html', - HAS_SOURCE: true, - SOURCELINK_SUFFIX: '.txt' - }; - </script> - <script type="text/javascript" src="_static/jquery.js"></script> - <script type="text/javascript" src="_static/underscore.js"></script> - <script type="text/javascript" src="_static/doctools.js"></script> - <script type="text/javascript" src="_static/searchtools.js"></script> + + + <script type="text/javascript" id="documentation_options" data-url_root="./" src="_static/documentation_options.js"></script> + <script type="text/javascript" src="_static/jquery.js"></script> + <script type="text/javascript" src="_static/underscore.js"></script> + <script type="text/javascript" src="_static/doctools.js"></script> + <script type="text/javascript" src="_static/searchtools.js"></script> +
