http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/executor/use-celery.html ---------------------------------------------------------------------- diff --git a/howto/executor/use-celery.html b/howto/executor/use-celery.html index 5305019..d29401f 100644 --- a/howto/executor/use-celery.html +++ b/howto/executor/use-celery.html @@ -24,28 +24,19 @@ - - - <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../../genindex.html"/> - <link rel="search" title="Search" href="../../search.html"/> - <link rel="top" title="Airflow Documentation" href="../../index.html"/> - <link rel="up" title="How-to Guides" href="../index.html"/> - <link rel="next" title="Scaling Out with Dask" href="use-dask.html"/> - <link rel="prev" title="Writing Logs" href="../write-logs.html"/> + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../../genindex.html" /> + <link rel="search" title="Search" href="../../search.html" /> + <link rel="next" title="Scaling Out with Dask" href="use-dask.html" /> + <link rel="prev" title="Writing Logs" href="../write-logs.html" /> <script src="../../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -96,6 +87,7 @@ <li class="toctree-l1 current"><a class="reference internal" href="../index.html">How-to Guides</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="../set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2"><a class="reference internal" href="../initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="../operator.html">Using Operators</a></li> <li class="toctree-l2"><a class="reference internal" href="../manage-connections.html">Managing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="../secure-connections.html">Securing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="../write-logs.html">Writing Logs</a></li> @@ -131,7 +123,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../../index.html">Airflow</a> @@ -139,9 +131,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -187,22 +180,22 @@ <div class="section" id="scaling-out-with-celery"> <h1>Scaling Out with Celery<a class="headerlink" href="#scaling-out-with-celery" title="Permalink to this headline">¶</a></h1> -<p><code class="docutils literal"><span class="pre">CeleryExecutor</span></code> is one of the ways you can scale out the number of workers. For this +<p><code class="docutils literal notranslate"><span class="pre">CeleryExecutor</span></code> is one of the ways you can scale out the number of workers. For this to work, you need to setup a Celery backend (<strong>RabbitMQ</strong>, <strong>Redis</strong>, â¦) and -change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point the executor parameter to -<code class="docutils literal"><span class="pre">CeleryExecutor</span></code> and provide the related Celery settings.</p> +change your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> to point the executor parameter to +<code class="docutils literal notranslate"><span class="pre">CeleryExecutor</span></code> and provide the related Celery settings.</p> <p>For more information about setting up a Celery broker, refer to the exhaustive <a class="reference external" href="http://docs.celeryproject.org/en/latest/getting-started/brokers/index.html";>Celery documentation on the topic</a>.</p> <p>Here are a few imperative requirements for your workers:</p> <ul class="simple"> -<li><code class="docutils literal"><span class="pre">airflow</span></code> needs to be installed, and the CLI needs to be in the path</li> +<li><code class="docutils literal notranslate"><span class="pre">airflow</span></code> needs to be installed, and the CLI needs to be in the path</li> <li>Airflow configuration settings should be homogeneous across the cluster</li> <li>Operators that are executed on the worker need to have their dependencies -met in that context. For example, if you use the <code class="docutils literal"><span class="pre">HiveOperator</span></code>, +met in that context. For example, if you use the <code class="docutils literal notranslate"><span class="pre">HiveOperator</span></code>, the hive CLI needs to be installed on that box, or if you use the -<code class="docutils literal"><span class="pre">MySqlOperator</span></code>, the required Python library needs to be available in -the <code class="docutils literal"><span class="pre">PYTHONPATH</span></code> somehow</li> -<li>The worker needs to have access to its <code class="docutils literal"><span class="pre">DAGS_FOLDER</span></code>, and you need to +<code class="docutils literal notranslate"><span class="pre">MySqlOperator</span></code>, the required Python library needs to be available in +the <code class="docutils literal notranslate"><span class="pre">PYTHONPATH</span></code> somehow</li> +<li>The worker needs to have access to its <code class="docutils literal notranslate"><span class="pre">DAGS_FOLDER</span></code>, and you need to synchronize the filesystems by your own means. A common setup would be to store your DAGS_FOLDER in a Git repository and sync it across machines using Chef, Puppet, Ansible, or whatever you use to configure machines in your @@ -211,13 +204,13 @@ pipelines files shared there should work as well</li> </ul> <p>To kick off a worker, you need to setup Airflow and kick off the worker subcommand</p> -<div class="highlight-bash"><div class="highlight"><pre><span></span>airflow worker +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>airflow worker </pre></div> </div> <p>Your worker should start picking up tasks as soon as they get fired in its direction.</p> <p>Note that you can also run âCelery Flowerâ, a web UI built on top of Celery, -to monitor your workers. You can use the shortcut command <code class="docutils literal"><span class="pre">airflow</span> <span class="pre">flower</span></code> +to monitor your workers. You can use the shortcut command <code class="docutils literal notranslate"><span class="pre">airflow</span> <span class="pre">flower</span></code> to start a Flower web server.</p> <p>Some caveats:</p> <ul class="simple"> @@ -229,9 +222,7 @@ to start a Flower web server.</p> </div> - <div class="articleComments"> - - </div> + </div> <footer> @@ -252,7 +243,7 @@ to start a Flower web server.</p> </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -271,6 +262,7 @@ to start a Flower web server.</p> var DOCUMENTATION_OPTIONS = { URL_ROOT:'../../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -283,19 +275,13 @@ to start a Flower web server.</p> - - - <script type="text/javascript" src="../../_static/js/theme.js"></script> - + <script type="text/javascript" src="../../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/executor/use-dask.html ---------------------------------------------------------------------- diff --git a/howto/executor/use-dask.html b/howto/executor/use-dask.html index d8b2380..bc18ef5 100644 --- a/howto/executor/use-dask.html +++ b/howto/executor/use-dask.html @@ -24,28 +24,19 @@ - - - <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../../genindex.html"/> - <link rel="search" title="Search" href="../../search.html"/> - <link rel="top" title="Airflow Documentation" href="../../index.html"/> - <link rel="up" title="How-to Guides" href="../index.html"/> - <link rel="next" title="Scaling Out with Mesos (community contributed)" href="use-mesos.html"/> - <link rel="prev" title="Scaling Out with Celery" href="use-celery.html"/> + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../../genindex.html" /> + <link rel="search" title="Search" href="../../search.html" /> + <link rel="next" title="Scaling Out with Mesos (community contributed)" href="use-mesos.html" /> + <link rel="prev" title="Scaling Out with Celery" href="use-celery.html" /> <script src="../../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -96,6 +87,7 @@ <li class="toctree-l1 current"><a class="reference internal" href="../index.html">How-to Guides</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="../set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2"><a class="reference internal" href="../initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="../operator.html">Using Operators</a></li> <li class="toctree-l2"><a class="reference internal" href="../manage-connections.html">Managing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="../secure-connections.html">Securing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="../write-logs.html">Writing Logs</a></li> @@ -131,7 +123,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../../index.html">Airflow</a> @@ -139,9 +131,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -187,11 +180,11 @@ <div class="section" id="scaling-out-with-dask"> <h1>Scaling Out with Dask<a class="headerlink" href="#scaling-out-with-dask" title="Permalink to this headline">¶</a></h1> -<p><code class="docutils literal"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p> +<p><code class="docutils literal notranslate"><span class="pre">DaskExecutor</span></code> allows you to run Airflow tasks in a Dask Distributed cluster.</p> <p>Dask clusters can be run on a single machine or on remote networks. For complete details, consult the <a class="reference external" href="https://distributed.readthedocs.io/";>Distributed documentation</a>.</p> <p>To create a cluster, first start a Scheduler:</p> -<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># default settings for a local cluster</span> <span class="nv">DASK_HOST</span><span class="o">=</span><span class="m">127</span>.0.0.1 <span class="nv">DASK_PORT</span><span class="o">=</span><span class="m">8786</span> @@ -199,11 +192,11 @@ dask-scheduler --host <span class="nv">$DASK_HOST</span> --port <span class="nv" </pre></div> </div> <p>Next start at least one Worker on any machine that can connect to the host:</p> -<div class="highlight-bash"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>dask-worker <span class="nv">$DASK_HOST</span>:<span class="nv">$DASK_PORT</span> </pre></div> </div> -<p>Edit your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal"><span class="pre">DaskExecutor</span></code> and provide -the Dask Scheduler address in the <code class="docutils literal"><span class="pre">[dask]</span></code> section.</p> +<p>Edit your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> to set your executor to <code class="docutils literal notranslate"><span class="pre">DaskExecutor</span></code> and provide +the Dask Scheduler address in the <code class="docutils literal notranslate"><span class="pre">[dask]</span></code> section.</p> <p>Please note:</p> <ul class="simple"> <li>Each Dask worker must be able to import Airflow and any dependencies you @@ -215,9 +208,7 @@ warning will be raised but the task will be submitted to the cluster.</li> </div> - <div class="articleComments"> - - </div> + </div> <footer> @@ -238,7 +229,7 @@ warning will be raised but the task will be submitted to the cluster.</li> </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -257,6 +248,7 @@ warning will be raised but the task will be submitted to the cluster.</li> var DOCUMENTATION_OPTIONS = { URL_ROOT:'../../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -269,19 +261,13 @@ warning will be raised but the task will be submitted to the cluster.</li> - - - <script type="text/javascript" src="../../_static/js/theme.js"></script> - + <script type="text/javascript" src="../../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/executor/use-mesos.html ---------------------------------------------------------------------- diff --git a/howto/executor/use-mesos.html b/howto/executor/use-mesos.html index f994908..ba905ea 100644 --- a/howto/executor/use-mesos.html +++ b/howto/executor/use-mesos.html @@ -24,28 +24,19 @@ - - - <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../../genindex.html"/> - <link rel="search" title="Search" href="../../search.html"/> - <link rel="top" title="Airflow Documentation" href="../../index.html"/> - <link rel="up" title="How-to Guides" href="../index.html"/> - <link rel="next" title="Running Airflow with systemd" href="../run-with-systemd.html"/> - <link rel="prev" title="Scaling Out with Dask" href="use-dask.html"/> + <link rel="stylesheet" href="../../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../../genindex.html" /> + <link rel="search" title="Search" href="../../search.html" /> + <link rel="next" title="Running Airflow with systemd" href="../run-with-systemd.html" /> + <link rel="prev" title="Scaling Out with Dask" href="use-dask.html" /> <script src="../../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -96,6 +87,7 @@ <li class="toctree-l1 current"><a class="reference internal" href="../index.html">How-to Guides</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="../set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2"><a class="reference internal" href="../initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="../operator.html">Using Operators</a></li> <li class="toctree-l2"><a class="reference internal" href="../manage-connections.html">Managing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="../secure-connections.html">Securing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="../write-logs.html">Writing Logs</a></li> @@ -135,7 +127,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../../index.html">Airflow</a> @@ -143,9 +135,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -198,7 +191,7 @@ </ol> <div class="section" id="tasks-executed-directly-on-mesos-slaves"> <h2>Tasks executed directly on mesos slaves<a class="headerlink" href="#tasks-executed-directly-on-mesos-slaves" title="Permalink to this headline">¶</a></h2> -<p><code class="docutils literal"><span class="pre">MesosExecutor</span></code> allows you to schedule airflow tasks on a Mesos cluster. +<p><code class="docutils literal notranslate"><span class="pre">MesosExecutor</span></code> allows you to schedule airflow tasks on a Mesos cluster. For this to work, you need a running mesos cluster and you must perform the following steps -</p> <ol class="arabic simple"> @@ -206,20 +199,20 @@ steps -</p> letâs refer to this as the âAirflow serverâ.</li> <li>On the Airflow server, install mesos python eggs from <a class="reference external" href="http://open.mesosphere.com/downloads/mesos/";>mesos downloads</a>.</li> <li>On the Airflow server, use a database (such as mysql) which can be accessed from all mesos -slaves and add configuration in <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</li> -<li>Change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point executor parameter to +slaves and add configuration in <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code>.</li> +<li>Change your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> to point executor parameter to <cite>MesosExecutor</cite> and provide related Mesos settings.</li> -<li>On all mesos slaves, install airflow. Copy the <code class="docutils literal"><span class="pre">airflow.cfg</span></code> from +<li>On all mesos slaves, install airflow. Copy the <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> from Airflow server (so that it uses same sql alchemy connection).</li> <li>On all mesos slaves, run the following for serving logs:</li> </ol> -<div class="highlight-bash"><div class="highlight"><pre><span></span>airflow serve_logs +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>airflow serve_logs </pre></div> </div> <ol class="arabic simple" start="7"> <li>On Airflow server, to start processing/scheduling DAGs on mesos, run:</li> </ol> -<div class="highlight-bash"><div class="highlight"><pre><span></span>airflow scheduler -p +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span>airflow scheduler -p </pre></div> </div> <p>Note: We need -p parameter to pickle the DAGs.</p> @@ -237,14 +230,14 @@ For any queries/bugs on <cite>MesosExecutor</cite>, please contact <a class="ref <blockquote> <div>We recommend taking advantage of dockerâs multi stage builds in order to achieve this. We have one Dockerfile that defines building a specific version of mesos from source (Dockerfile-mesos), in order to create the python eggs. In the airflow Dockerfile (Dockerfile-airflow) we copy the python eggs from the mesos image.</div></blockquote> <ol class="arabic simple" start="2"> -<li>Create a mesos configuration block within the <code class="docutils literal"><span class="pre">airflow.cfg</span></code>.</li> +<li>Create a mesos configuration block within the <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code>.</li> </ol> <blockquote> -<div>The configuration block remains the same as the default airflow configuration (default_airflow.cfg), but has the addition of an option <code class="docutils literal"><span class="pre">docker_image_slave</span></code>. This should be set to the name of the image you would like mesos to use when running airflow tasks. Make sure you have the proper configuration of the DNS record for your mesos master and any sort of authorization if any exists.</div></blockquote> +<div>The configuration block remains the same as the default airflow configuration (default_airflow.cfg), but has the addition of an option <code class="docutils literal notranslate"><span class="pre">docker_image_slave</span></code>. This should be set to the name of the image you would like mesos to use when running airflow tasks. Make sure you have the proper configuration of the DNS record for your mesos master and any sort of authorization if any exists.</div></blockquote> <ol class="arabic simple" start="3"> -<li>Change your <code class="docutils literal"><span class="pre">airflow.cfg</span></code> to point the executor parameter to +<li>Change your <code class="docutils literal notranslate"><span class="pre">airflow.cfg</span></code> to point the executor parameter to <cite>MesosExecutor</cite> (<cite>executor = SequentialExecutor</cite>).</li> -<li>Make sure your mesos slave has access to the docker repository you are using for your <code class="docutils literal"><span class="pre">docker_image_slave</span></code>.</li> +<li>Make sure your mesos slave has access to the docker repository you are using for your <code class="docutils literal notranslate"><span class="pre">docker_image_slave</span></code>.</li> </ol> <blockquote> <div><a class="reference external" href="https://mesos.readthedocs.io/en/latest/docker-containerizer/#private-docker-repository";>Instructions are available in the mesos docs.</a></div></blockquote> @@ -254,9 +247,7 @@ For any queries/bugs on <cite>MesosExecutor</cite>, please contact <a class="ref </div> - <div class="articleComments"> - - </div> + </div> <footer> @@ -277,7 +268,7 @@ For any queries/bugs on <cite>MesosExecutor</cite>, please contact <a class="ref </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -296,6 +287,7 @@ For any queries/bugs on <cite>MesosExecutor</cite>, please contact <a class="ref var DOCUMENTATION_OPTIONS = { URL_ROOT:'../../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -308,19 +300,13 @@ For any queries/bugs on <cite>MesosExecutor</cite>, please contact <a class="ref - - - <script type="text/javascript" src="../../_static/js/theme.js"></script> - + <script type="text/javascript" src="../../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/index.html ---------------------------------------------------------------------- diff --git a/howto/index.html b/howto/index.html index 125a05f..4189bc4 100644 --- a/howto/index.html +++ b/howto/index.html @@ -24,27 +24,19 @@ - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="Airflow Documentation" href="../index.html"/> - <link rel="next" title="Setting Configuration Options" href="set-config.html"/> - <link rel="prev" title="Tutorial" href="../tutorial.html"/> + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + <link rel="next" title="Setting Configuration Options" href="set-config.html" /> + <link rel="prev" title="Tutorial" href="../tutorial.html" /> <script src="../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -95,6 +87,7 @@ <li class="toctree-l1 current"><a class="current reference internal" href="#">How-to Guides</a><ul> <li class="toctree-l2"><a class="reference internal" href="set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2"><a class="reference internal" href="initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="operator.html">Using Operators</a></li> <li class="toctree-l2"><a class="reference internal" href="manage-connections.html">Managing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="secure-connections.html">Securing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="write-logs.html">Writing Logs</a></li> @@ -130,7 +123,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../index.html">Airflow</a> @@ -138,9 +131,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -192,6 +186,12 @@ configuring an Airflow environment.</p> <ul> <li class="toctree-l1"><a class="reference internal" href="set-config.html">Setting Configuration Options</a></li> <li class="toctree-l1"><a class="reference internal" href="initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l1"><a class="reference internal" href="operator.html">Using Operators</a><ul> +<li class="toctree-l2"><a class="reference internal" href="operator.html#bashoperator">BashOperator</a></li> +<li class="toctree-l2"><a class="reference internal" href="operator.html#pythonoperator">PythonOperator</a></li> +<li class="toctree-l2"><a class="reference internal" href="operator.html#google-cloud-platform-operators">Google Cloud Platform Operators</a></li> +</ul> +</li> <li class="toctree-l1"><a class="reference internal" href="manage-connections.html">Managing Connections</a><ul> <li class="toctree-l2"><a class="reference internal" href="manage-connections.html#creating-a-connection-with-the-ui">Creating a Connection with the UI</a></li> <li class="toctree-l2"><a class="reference internal" href="manage-connections.html#editing-a-connection-with-the-ui">Editing a Connection with the UI</a></li> @@ -223,9 +223,7 @@ configuring an Airflow environment.</p> </div> - <div class="articleComments"> - - </div> + </div> <footer> @@ -246,7 +244,7 @@ configuring an Airflow environment.</p> </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -265,6 +263,7 @@ configuring an Airflow environment.</p> var DOCUMENTATION_OPTIONS = { URL_ROOT:'../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -277,19 +276,13 @@ configuring an Airflow environment.</p> - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - + <script type="text/javascript" src="../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/initialize-database.html ---------------------------------------------------------------------- diff --git a/howto/initialize-database.html b/howto/initialize-database.html index 66edb0c..b03d757 100644 --- a/howto/initialize-database.html +++ b/howto/initialize-database.html @@ -24,28 +24,19 @@ - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="Airflow Documentation" href="../index.html"/> - <link rel="up" title="How-to Guides" href="index.html"/> - <link rel="next" title="Managing Connections" href="manage-connections.html"/> - <link rel="prev" title="Setting Configuration Options" href="set-config.html"/> + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + <link rel="next" title="Using Operators" href="operator.html" /> + <link rel="prev" title="Setting Configuration Options" href="set-config.html" /> <script src="../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -96,6 +87,7 @@ <li class="toctree-l1 current"><a class="reference internal" href="index.html">How-to Guides</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2 current"><a class="current reference internal" href="#">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="operator.html">Using Operators</a></li> <li class="toctree-l2"><a class="reference internal" href="manage-connections.html">Managing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="secure-connections.html">Securing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="write-logs.html">Writing Logs</a></li> @@ -131,7 +123,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../index.html">Airflow</a> @@ -139,9 +131,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -200,19 +193,19 @@ in your my.cnf under <cite>[mysqld]</cite></p> </div> <div class="admonition note"> <p class="first admonition-title">Note</p> -<p class="last">If you decide to use <strong>Postgres</strong>, we recommend using the <code class="docutils literal"><span class="pre">psycopg2</span></code> +<p class="last">If you decide to use <strong>Postgres</strong>, we recommend using the <code class="docutils literal notranslate"><span class="pre">psycopg2</span></code> driver and specifying it in your SqlAlchemy connection string. Also note that since SqlAlchemy does not expose a way to target a specific schema in the Postgres connection URI, you may want to set a default schema for your role with a -command similar to <code class="docutils literal"><span class="pre">ALTER</span> <span class="pre">ROLE</span> <span class="pre">username</span> <span class="pre">SET</span> <span class="pre">search_path</span> <span class="pre">=</span> <span class="pre">airflow,</span> <span class="pre">foobar;</span></code></p> +command similar to <code class="docutils literal notranslate"><span class="pre">ALTER</span> <span class="pre">ROLE</span> <span class="pre">username</span> <span class="pre">SET</span> <span class="pre">search_path</span> <span class="pre">=</span> <span class="pre">airflow,</span> <span class="pre">foobar;</span></code></p> </div> <p>Once youâve setup your database to host Airflow, youâll need to alter the SqlAlchemy connection string located in your configuration file -<code class="docutils literal"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the âexecutorâ +<code class="docutils literal notranslate"><span class="pre">$AIRFLOW_HOME/airflow.cfg</span></code>. You should then also change the âexecutorâ setting to use âLocalExecutorâ, an executor that can parallelize task instances locally.</p> -<div class="highlight-bash"><div class="highlight"><pre><span></span><span class="c1"># initialize the database</span> +<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="c1"># initialize the database</span> airflow initdb </pre></div> </div> @@ -220,15 +213,13 @@ airflow initdb </div> - <div class="articleComments"> - - </div> + </div> <footer> <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> - <a href="manage-connections.html" class="btn btn-neutral float-right" title="Managing Connections" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> + <a href="operator.html" class="btn btn-neutral float-right" title="Using Operators" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> <a href="set-config.html" class="btn btn-neutral" title="Setting Configuration Options" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> @@ -243,7 +234,7 @@ airflow initdb </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -262,6 +253,7 @@ airflow initdb var DOCUMENTATION_OPTIONS = { URL_ROOT:'../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -274,19 +266,13 @@ airflow initdb - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - + <script type="text/javascript" src="../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/manage-connections.html ---------------------------------------------------------------------- diff --git a/howto/manage-connections.html b/howto/manage-connections.html index 62547ee..0cedef4 100644 --- a/howto/manage-connections.html +++ b/howto/manage-connections.html @@ -24,28 +24,19 @@ - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="Airflow Documentation" href="../index.html"/> - <link rel="up" title="How-to Guides" href="index.html"/> - <link rel="next" title="Securing Connections" href="secure-connections.html"/> - <link rel="prev" title="Initializing a Database Backend" href="initialize-database.html"/> + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + <link rel="next" title="Securing Connections" href="secure-connections.html" /> + <link rel="prev" title="Using Operators" href="operator.html" /> <script src="../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -96,6 +87,7 @@ <li class="toctree-l1 current"><a class="reference internal" href="index.html">How-to Guides</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2"><a class="reference internal" href="initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="operator.html">Using Operators</a></li> <li class="toctree-l2 current"><a class="current reference internal" href="#">Managing Connections</a><ul> <li class="toctree-l3"><a class="reference internal" href="#creating-a-connection-with-the-ui">Creating a Connection with the UI</a></li> <li class="toctree-l3"><a class="reference internal" href="#editing-a-connection-with-the-ui">Editing a Connection with the UI</a></li> @@ -140,7 +132,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../index.html">Airflow</a> @@ -148,9 +140,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -198,7 +191,7 @@ <h1>Managing Connections<a class="headerlink" href="#managing-connections" title="Permalink to this headline">¶</a></h1> <p>Airflow needs to know how to connect to your environment. Information such as hostname, port, login and passwords to other systems and services is -handled in the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. The pipeline code you +handled in the <code class="docutils literal notranslate"><span class="pre">Admin->Connection</span></code> section of the UI. The pipeline code you will author will reference the âconn_idâ of the Connection objects.</p> <img alt="../_images/connections.png" src="../_images/connections.png" /> <p>Connections can be created and managed using either the UI or environment @@ -207,41 +200,41 @@ variables.</p> more information.</p> <div class="section" id="creating-a-connection-with-the-ui"> <h2>Creating a Connection with the UI<a class="headerlink" href="#creating-a-connection-with-the-ui" title="Permalink to this headline">¶</a></h2> -<p>Open the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. Click the <code class="docutils literal"><span class="pre">Create</span></code> link +<p>Open the <code class="docutils literal notranslate"><span class="pre">Admin->Connection</span></code> section of the UI. Click the <code class="docutils literal notranslate"><span class="pre">Create</span></code> link to create a new connection.</p> <img alt="../_images/connection_create.png" src="../_images/connection_create.png" /> <ol class="arabic simple"> -<li>Fill in the <code class="docutils literal"><span class="pre">Conn</span> <span class="pre">Id</span></code> field with the desired connection ID. It is +<li>Fill in the <code class="docutils literal notranslate"><span class="pre">Conn</span> <span class="pre">Id</span></code> field with the desired connection ID. It is recommended that you use lower-case characters and separate words with underscores.</li> -<li>Choose the connection type with the <code class="docutils literal"><span class="pre">Conn</span> <span class="pre">Type</span></code> field.</li> +<li>Choose the connection type with the <code class="docutils literal notranslate"><span class="pre">Conn</span> <span class="pre">Type</span></code> field.</li> <li>Fill in the remaining fields. See <a class="reference internal" href="#manage-connections-connection-types"><span class="std std-ref">Connection Types</span></a> for a description of the fields belonging to the different connection types.</li> -<li>Click the <code class="docutils literal"><span class="pre">Save</span></code> button to create the connection.</li> +<li>Click the <code class="docutils literal notranslate"><span class="pre">Save</span></code> button to create the connection.</li> </ol> </div> <div class="section" id="editing-a-connection-with-the-ui"> <h2>Editing a Connection with the UI<a class="headerlink" href="#editing-a-connection-with-the-ui" title="Permalink to this headline">¶</a></h2> -<p>Open the <code class="docutils literal"><span class="pre">Admin->Connection</span></code> section of the UI. Click the pencil icon next +<p>Open the <code class="docutils literal notranslate"><span class="pre">Admin->Connection</span></code> section of the UI. Click the pencil icon next to the connection you wish to edit in the connection list.</p> <img alt="../_images/connection_edit.png" src="../_images/connection_edit.png" /> -<p>Modify the connection properties and click the <code class="docutils literal"><span class="pre">Save</span></code> button to save your +<p>Modify the connection properties and click the <code class="docutils literal notranslate"><span class="pre">Save</span></code> button to save your changes.</p> </div> <div class="section" id="creating-a-connection-with-environment-variables"> <h2>Creating a Connection with Environment Variables<a class="headerlink" href="#creating-a-connection-with-environment-variables" title="Permalink to this headline">¶</a></h2> <p>Connections in Airflow pipelines can be created using environment variables. -The environment variable needs to have a prefix of <code class="docutils literal"><span class="pre">AIRFLOW_CONN_</span></code> for +The environment variable needs to have a prefix of <code class="docutils literal notranslate"><span class="pre">AIRFLOW_CONN_</span></code> for Airflow with the value in a URI format to use the connection properly.</p> -<p>When referencing the connection in the Airflow pipeline, the <code class="docutils literal"><span class="pre">conn_id</span></code> +<p>When referencing the connection in the Airflow pipeline, the <code class="docutils literal notranslate"><span class="pre">conn_id</span></code> should be the name of the variable without the prefix. For example, if the -<code class="docutils literal"><span class="pre">conn_id</span></code> is named <code class="docutils literal"><span class="pre">postgres_master</span></code> the environment variable should be -named <code class="docutils literal"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code> (note that the environment variable +<code class="docutils literal notranslate"><span class="pre">conn_id</span></code> is named <code class="docutils literal notranslate"><span class="pre">postgres_master</span></code> the environment variable should be +named <code class="docutils literal notranslate"><span class="pre">AIRFLOW_CONN_POSTGRES_MASTER</span></code> (note that the environment variable must be all uppercase). Airflow assumes the value returned from the environment variable to be in a URI format (e.g. -<code class="docutils literal"><span class="pre">postgres://user:password@localhost:5432/master</span></code> or -<code class="docutils literal"><span class="pre">s3://accesskey:secretkey@S3</span></code>).</p> +<code class="docutils literal notranslate"><span class="pre">postgres://user:password@localhost:5432/master</span></code> or +<code class="docutils literal notranslate"><span class="pre">s3://accesskey:secretkey@S3</span></code>).</p> </div> <div class="section" id="connection-types"> <span id="manage-connections-connection-types"></span><h2>Connection Types<a class="headerlink" href="#connection-types" title="Permalink to this headline">¶</a></h2> @@ -262,19 +255,19 @@ file (JSON format) on disk.</li> <h4>Default Connection IDs<a class="headerlink" href="#default-connection-ids" title="Permalink to this headline">¶</a></h4> <p>The following connection IDs are used by default.</p> <dl class="docutils"> -<dt><code class="docutils literal"><span class="pre">bigquery_default</span></code></dt> -<dd>Used by the <a class="reference internal" href="../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook" title="airflow.contrib.hooks.bigquery_hook.BigQueryHook"><code class="xref py py-class docutils literal"><span class="pre">BigQueryHook</span></code></a> +<dt><code class="docutils literal notranslate"><span class="pre">bigquery_default</span></code></dt> +<dd>Used by the <a class="reference internal" href="../integration.html#airflow.contrib.hooks.bigquery_hook.BigQueryHook" title="airflow.contrib.hooks.bigquery_hook.BigQueryHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">BigQueryHook</span></code></a> hook.</dd> -<dt><code class="docutils literal"><span class="pre">google_cloud_datastore_default</span></code></dt> -<dd>Used by the <a class="reference internal" href="../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook" title="airflow.contrib.hooks.datastore_hook.DatastoreHook"><code class="xref py py-class docutils literal"><span class="pre">DatastoreHook</span></code></a> +<dt><code class="docutils literal notranslate"><span class="pre">google_cloud_datastore_default</span></code></dt> +<dd>Used by the <a class="reference internal" href="../integration.html#airflow.contrib.hooks.datastore_hook.DatastoreHook" title="airflow.contrib.hooks.datastore_hook.DatastoreHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">DatastoreHook</span></code></a> hook.</dd> -<dt><code class="docutils literal"><span class="pre">google_cloud_default</span></code></dt> +<dt><code class="docutils literal notranslate"><span class="pre">google_cloud_default</span></code></dt> <dd>Used by the -<a class="reference internal" href="../code.html#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal"><span class="pre">GoogleCloudBaseHook</span></code></a>, -<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook" title="airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook"><code class="xref py py-class docutils literal"><span class="pre">DataFlowHook</span></code></a>, -<a class="reference internal" href="../code.html#airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook" title="airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook"><code class="xref py py-class docutils literal"><span class="pre">DataProcHook</span></code></a>, -<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook" title="airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook"><code class="xref py py-class docutils literal"><span class="pre">MLEngineHook</span></code></a>, and -<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook"><code class="xref py py-class docutils literal"><span class="pre">GoogleCloudStorageHook</span></code></a> hooks.</dd> +<a class="reference internal" href="../code.html#airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook" title="airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">GoogleCloudBaseHook</span></code></a>, +<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook" title="airflow.contrib.hooks.gcp_dataflow_hook.DataFlowHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataFlowHook</span></code></a>, +<a class="reference internal" href="../code.html#airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook" title="airflow.contrib.hooks.gcp_dataproc_hook.DataProcHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">DataProcHook</span></code></a>, +<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook" title="airflow.contrib.hooks.gcp_mlengine_hook.MLEngineHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">MLEngineHook</span></code></a>, and +<a class="reference internal" href="../integration.html#airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook" title="airflow.contrib.hooks.gcs_hook.GoogleCloudStorageHook"><code class="xref py py-class docutils literal notranslate"><span class="pre">GoogleCloudStorageHook</span></code></a> hooks.</dd> </dl> </div> <div class="section" id="configuring-the-connection"> @@ -309,9 +302,7 @@ issue <a class="reference external" href="https://issues.apache.org/jira/browse/ </div> - <div class="articleComments"> - - </div> + </div> <footer> @@ -320,7 +311,7 @@ issue <a class="reference external" href="https://issues.apache.org/jira/browse/ <a href="secure-connections.html" class="btn btn-neutral float-right" title="Securing Connections" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> - <a href="initialize-database.html" class="btn btn-neutral" title="Initializing a Database Backend" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> + <a href="operator.html" class="btn btn-neutral" title="Using Operators" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> </div> @@ -332,7 +323,7 @@ issue <a class="reference external" href="https://issues.apache.org/jira/browse/ </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -351,6 +342,7 @@ issue <a class="reference external" href="https://issues.apache.org/jira/browse/ var DOCUMENTATION_OPTIONS = { URL_ROOT:'../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -363,19 +355,13 @@ issue <a class="reference external" href="https://issues.apache.org/jira/browse/ - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - + <script type="text/javascript" src="../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/operator.html ---------------------------------------------------------------------- diff --git a/howto/operator.html b/howto/operator.html new file mode 100644 index 0000000..e2605df --- /dev/null +++ b/howto/operator.html @@ -0,0 +1,400 @@ + + +<!DOCTYPE html> +<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]--> +<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]--> +<head> + <meta charset="utf-8"> + + <meta name="viewport" content="width=device-width, initial-scale=1.0"> + + <title>Using Operators — Airflow Documentation</title> + + + + + + + + + + + + + + + + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + <link rel="next" title="Managing Connections" href="manage-connections.html" /> + <link rel="prev" title="Initializing a Database Backend" href="initialize-database.html" /> + + + <script src="../_static/js/modernizr.min.js"></script> + +</head> + +<body class="wy-body-for-nav"> + + + <div class="wy-grid-for-nav"> + + + <nav data-toggle="wy-nav-shift" class="wy-nav-side"> + <div class="wy-side-scroll"> + <div class="wy-side-nav-search"> + + + + <a href="../index.html" class="icon icon-home"> Airflow + + + + </a> + + + + + + + +<div role="search"> + <form id="rtd-search-form" class="wy-form" action="../search.html" method="get"> + <input type="text" name="q" placeholder="Search docs" /> + <input type="hidden" name="check_keywords" value="yes" /> + <input type="hidden" name="area" value="default" /> + </form> +</div> + + + </div> + + <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation"> + + + + + + + <ul class="current"> +<li class="toctree-l1"><a class="reference internal" href="../project.html">Project</a></li> +<li class="toctree-l1"><a class="reference internal" href="../license.html">License</a></li> +<li class="toctree-l1"><a class="reference internal" href="../start.html">Quick Start</a></li> +<li class="toctree-l1"><a class="reference internal" href="../installation.html">Installation</a></li> +<li class="toctree-l1"><a class="reference internal" href="../tutorial.html">Tutorial</a></li> +<li class="toctree-l1 current"><a class="reference internal" href="index.html">How-to Guides</a><ul class="current"> +<li class="toctree-l2"><a class="reference internal" href="set-config.html">Setting Configuration Options</a></li> +<li class="toctree-l2"><a class="reference internal" href="initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2 current"><a class="current reference internal" href="#">Using Operators</a><ul> +<li class="toctree-l3"><a class="reference internal" href="#bashoperator">BashOperator</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#templating">Templating</a></li> +<li class="toctree-l4"><a class="reference internal" href="#troubleshooting">Troubleshooting</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="#pythonoperator">PythonOperator</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#passing-in-arguments">Passing in arguments</a></li> +<li class="toctree-l4"><a class="reference internal" href="#id1">Templating</a></li> +</ul> +</li> +<li class="toctree-l3"><a class="reference internal" href="#google-cloud-platform-operators">Google Cloud Platform Operators</a><ul> +<li class="toctree-l4"><a class="reference internal" href="#googlecloudstoragetobigqueryoperator">GoogleCloudStorageToBigQueryOperator</a></li> +</ul> +</li> +</ul> +</li> +<li class="toctree-l2"><a class="reference internal" href="manage-connections.html">Managing Connections</a></li> +<li class="toctree-l2"><a class="reference internal" href="secure-connections.html">Securing Connections</a></li> +<li class="toctree-l2"><a class="reference internal" href="write-logs.html">Writing Logs</a></li> +<li class="toctree-l2"><a class="reference internal" href="executor/use-celery.html">Scaling Out with Celery</a></li> +<li class="toctree-l2"><a class="reference internal" href="executor/use-dask.html">Scaling Out with Dask</a></li> +<li class="toctree-l2"><a class="reference internal" href="executor/use-mesos.html">Scaling Out with Mesos (community contributed)</a></li> +<li class="toctree-l2"><a class="reference internal" href="run-with-systemd.html">Running Airflow with systemd</a></li> +<li class="toctree-l2"><a class="reference internal" href="run-with-upstart.html">Running Airflow with upstart</a></li> +<li class="toctree-l2"><a class="reference internal" href="use-test-config.html">Using the Test Mode Configuration</a></li> +</ul> +</li> +<li class="toctree-l1"><a class="reference internal" href="../ui.html">UI / Screenshots</a></li> +<li class="toctree-l1"><a class="reference internal" href="../concepts.html">Concepts</a></li> +<li class="toctree-l1"><a class="reference internal" href="../profiling.html">Data Profiling</a></li> +<li class="toctree-l1"><a class="reference internal" href="../cli.html">Command Line Interface</a></li> +<li class="toctree-l1"><a class="reference internal" href="../scheduler.html">Scheduling & Triggers</a></li> +<li class="toctree-l1"><a class="reference internal" href="../plugins.html">Plugins</a></li> +<li class="toctree-l1"><a class="reference internal" href="../security.html">Security</a></li> +<li class="toctree-l1"><a class="reference internal" href="../timezone.html">Time zones</a></li> +<li class="toctree-l1"><a class="reference internal" href="../api.html">Experimental Rest API</a></li> +<li class="toctree-l1"><a class="reference internal" href="../integration.html">Integration</a></li> +<li class="toctree-l1"><a class="reference internal" href="../lineage.html">Lineage</a></li> +<li class="toctree-l1"><a class="reference internal" href="../faq.html">FAQ</a></li> +<li class="toctree-l1"><a class="reference internal" href="../code.html">API Reference</a></li> +</ul> + + + + </div> + </div> + </nav> + + <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> + + + <nav class="wy-nav-top" aria-label="top navigation"> + + <i data-toggle="wy-nav-top" class="fa fa-bars"></i> + <a href="../index.html">Airflow</a> + + </nav> + + + <div class="wy-nav-content"> + + <div class="rst-content"> + + + + + + + + + + + + + + + + + +<div role="navigation" aria-label="breadcrumbs navigation"> + + <ul class="wy-breadcrumbs"> + + <li><a href="../index.html">Docs</a> »</li> + + <li><a href="index.html">How-to Guides</a> »</li> + + <li>Using Operators</li> + + + <li class="wy-breadcrumbs-aside"> + + + <a href="../_sources/howto/operator.rst.txt" rel="nofollow"> View page source</a> + + + </li> + + </ul> + + + <hr/> +</div> + <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article";> + <div itemprop="articleBody"> + + <div class="section" id="using-operators"> +<h1>Using Operators<a class="headerlink" href="#using-operators" title="Permalink to this headline">¶</a></h1> +<p>An operator represents a single, ideally idempotent, task. Operators +determine what actually executes when your DAG runs.</p> +<p>See the <a class="reference internal" href="../concepts.html#concepts-operators"><span class="std std-ref">Operators Concepts</span></a> documentation and the +<a class="reference internal" href="../code.html#api-reference-operators"><span class="std std-ref">Operators API Reference</span></a> for more +information.</p> +<div class="contents local topic" id="contents"> +<ul class="simple"> +<li><a class="reference internal" href="#bashoperator" id="id2">BashOperator</a><ul> +<li><a class="reference internal" href="#templating" id="id3">Templating</a></li> +<li><a class="reference internal" href="#troubleshooting" id="id4">Troubleshooting</a><ul> +<li><a class="reference internal" href="#jinja-template-not-found" id="id5">Jinja template not found</a></li> +</ul> +</li> +</ul> +</li> +<li><a class="reference internal" href="#pythonoperator" id="id6">PythonOperator</a><ul> +<li><a class="reference internal" href="#passing-in-arguments" id="id7">Passing in arguments</a></li> +<li><a class="reference internal" href="#id1" id="id8">Templating</a></li> +</ul> +</li> +<li><a class="reference internal" href="#google-cloud-platform-operators" id="id9">Google Cloud Platform Operators</a><ul> +<li><a class="reference internal" href="#googlecloudstoragetobigqueryoperator" id="id10">GoogleCloudStorageToBigQueryOperator</a></li> +</ul> +</li> +</ul> +</div> +<div class="section" id="bashoperator"> +<h2><a class="toc-backref" href="#id2">BashOperator</a><a class="headerlink" href="#bashoperator" title="Permalink to this headline">¶</a></h2> +<p>Use the <a class="reference internal" href="../code.html#airflow.operators.bash_operator.BashOperator" title="airflow.operators.bash_operator.BashOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">BashOperator</span></code></a> to execute +commands in a <a class="reference external" href="https://www.gnu.org/software/bash/";>Bash</a> shell.</p> +<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">run_this</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span> + <span class="n">task_id</span><span class="o">=</span><span class="s1">'run_after_loop'</span><span class="p">,</span> <span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo 1'</span><span class="p">,</span> <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +</pre></div> +</div> +<div class="section" id="templating"> +<h3><a class="toc-backref" href="#id3">Templating</a><a class="headerlink" href="#templating" title="Permalink to this headline">¶</a></h3> +<p>You can use <a class="reference internal" href="../concepts.html#jinja-templating"><span class="std std-ref">Jinja templates</span></a> to parameterize the +<code class="docutils literal notranslate"><span class="pre">bash_command</span></code> argument.</p> +<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">task</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span> + <span class="n">task_id</span><span class="o">=</span><span class="s1">'also_run_this'</span><span class="p">,</span> + <span class="n">bash_command</span><span class="o">=</span><span class="s1">'echo "run_id={{ run_id }} | dag_run={{ dag_run }}"'</span><span class="p">,</span> + <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +</pre></div> +</div> +</div> +<div class="section" id="troubleshooting"> +<h3><a class="toc-backref" href="#id4">Troubleshooting</a><a class="headerlink" href="#troubleshooting" title="Permalink to this headline">¶</a></h3> +<div class="section" id="jinja-template-not-found"> +<h4><a class="toc-backref" href="#id5">Jinja template not found</a><a class="headerlink" href="#jinja-template-not-found" title="Permalink to this headline">¶</a></h4> +<p>Add a space after the script name when directly calling a Bash script with +the <code class="docutils literal notranslate"><span class="pre">bash_command</span></code> argument. This is because Airflow tries to apply a Jinja +template to it, which will fail.</p> +<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">t2</span> <span class="o">=</span> <span class="n">BashOperator</span><span class="p">(</span> + <span class="n">task_id</span><span class="o">=</span><span class="s1">'bash_example'</span><span class="p">,</span> + + <span class="c1"># This fails with `Jinja template not found` error</span> + <span class="c1"># bash_command="/home/batcher/test.sh",</span> + + <span class="c1"># This works (has a space after)</span> + <span class="n">bash_command</span><span class="o">=</span><span class="s2">"/home/batcher/test.sh "</span><span class="p">,</span> + <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +</pre></div> +</div> +</div> +</div> +</div> +<div class="section" id="pythonoperator"> +<h2><a class="toc-backref" href="#id6">PythonOperator</a><a class="headerlink" href="#pythonoperator" title="Permalink to this headline">¶</a></h2> +<p>Use the <a class="reference internal" href="../code.html#airflow.operators.python_operator.PythonOperator" title="airflow.operators.python_operator.PythonOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">PythonOperator</span></code></a> to execute +Python callables.</p> +<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">print_context</span><span class="p">(</span><span class="n">ds</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span> + <span class="n">pprint</span><span class="p">(</span><span class="n">kwargs</span><span class="p">)</span> + <span class="k">print</span><span class="p">(</span><span class="n">ds</span><span class="p">)</span> + <span class="k">return</span> <span class="s1">'Whatever you return gets printed in the logs'</span> + + +<span class="n">run_this</span> <span class="o">=</span> <span class="n">PythonOperator</span><span class="p">(</span> + <span class="n">task_id</span><span class="o">=</span><span class="s1">'print_the_context'</span><span class="p">,</span> + <span class="n">provide_context</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> + <span class="n">python_callable</span><span class="o">=</span><span class="n">print_context</span><span class="p">,</span> + <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +</pre></div> +</div> +<div class="section" id="passing-in-arguments"> +<h3><a class="toc-backref" href="#id7">Passing in arguments</a><a class="headerlink" href="#passing-in-arguments" title="Permalink to this headline">¶</a></h3> +<p>Use the <code class="docutils literal notranslate"><span class="pre">op_args</span></code> and <code class="docutils literal notranslate"><span class="pre">op_kwargs</span></code> arguments to pass additional arguments +to the Python callable.</p> +<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">my_sleeping_function</span><span class="p">(</span><span class="n">random_base</span><span class="p">):</span> + <span class="sd">"""This is a function that will run within the DAG execution"""</span> + <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="n">random_base</span><span class="p">)</span> + + +<span class="c1"># Generate 10 sleeping tasks, sleeping from 0 to 4 seconds respectively</span> +<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">5</span><span class="p">):</span> + <span class="n">task</span> <span class="o">=</span> <span class="n">PythonOperator</span><span class="p">(</span> + <span class="n">task_id</span><span class="o">=</span><span class="s1">'sleep_for_'</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="n">i</span><span class="p">),</span> + <span class="n">python_callable</span><span class="o">=</span><span class="n">my_sleeping_function</span><span class="p">,</span> + <span class="n">op_kwargs</span><span class="o">=</span><span class="p">{</span><span class="s1">'random_base'</span><span class="p">:</span> <span class="nb">float</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="o">/</span> <span class="mi">10</span><span class="p">},</span> + <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> + + <span class="n">task</span><span class="o">.</span><span class="n">set_upstream</span><span class="p">(</span><span class="n">run_this</span><span class="p">)</span> +</pre></div> +</div> +</div> +<div class="section" id="id1"> +<h3><a class="toc-backref" href="#id8">Templating</a><a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h3> +<p>When you set the <code class="docutils literal notranslate"><span class="pre">provide_context</span></code> argument to <code class="docutils literal notranslate"><span class="pre">True</span></code>, Airflow passes in +an additional set of keyword arguments: one for each of the <a class="reference internal" href="../code.html#macros"><span class="std std-ref">Jinja +template variables</span></a> and a <code class="docutils literal notranslate"><span class="pre">templates_dict</span></code> argument.</p> +<p>The <code class="docutils literal notranslate"><span class="pre">templates_dict</span></code> argument is templated, so each value in the dictionary +is evaluated as a <a class="reference internal" href="../concepts.html#jinja-templating"><span class="std std-ref">Jinja template</span></a>.</p> +</div> +</div> +<div class="section" id="google-cloud-platform-operators"> +<h2><a class="toc-backref" href="#id9">Google Cloud Platform Operators</a><a class="headerlink" href="#google-cloud-platform-operators" title="Permalink to this headline">¶</a></h2> +<div class="section" id="googlecloudstoragetobigqueryoperator"> +<h3><a class="toc-backref" href="#id10">GoogleCloudStorageToBigQueryOperator</a><a class="headerlink" href="#googlecloudstoragetobigqueryoperator" title="Permalink to this headline">¶</a></h3> +<p>Use the +<a class="reference internal" href="../integration.html#airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator" title="airflow.contrib.operators.gcs_to_bq.GoogleCloudStorageToBigQueryOperator"><code class="xref py py-class docutils literal notranslate"><span class="pre">GoogleCloudStorageToBigQueryOperator</span></code></a> +to execute a BigQuery load job.</p> +<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="n">load_csv</span> <span class="o">=</span> <span class="n">gcs_to_bq</span><span class="o">.</span><span class="n">GoogleCloudStorageToBigQueryOperator</span><span class="p">(</span> + <span class="n">task_id</span><span class="o">=</span><span class="s1">'gcs_to_bq_example'</span><span class="p">,</span> + <span class="n">bucket</span><span class="o">=</span><span class="s1">'cloud-samples-data'</span><span class="p">,</span> + <span class="n">source_objects</span><span class="o">=</span><span class="p">[</span><span class="s1">'bigquery/us-states/us-states.csv'</span><span class="p">],</span> + <span class="n">destination_project_dataset_table</span><span class="o">=</span><span class="s1">'airflow_test.gcs_to_bq_table'</span><span class="p">,</span> + <span class="n">schema_fields</span><span class="o">=</span><span class="p">[</span> + <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'name'</span><span class="p">,</span> <span class="s1">'type'</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'mode'</span><span class="p">:</span> <span class="s1">'NULLABLE'</span><span class="p">},</span> + <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'post_abbr'</span><span class="p">,</span> <span class="s1">'type'</span><span class="p">:</span> <span class="s1">'STRING'</span><span class="p">,</span> <span class="s1">'mode'</span><span class="p">:</span> <span class="s1">'NULLABLE'</span><span class="p">},</span> + <span class="p">],</span> + <span class="n">write_disposition</span><span class="o">=</span><span class="s1">'WRITE_TRUNCATE'</span><span class="p">,</span> + <span class="n">dag</span><span class="o">=</span><span class="n">dag</span><span class="p">)</span> +</pre></div> +</div> +</div> +</div> +</div> + + + </div> + + </div> + <footer> + + <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation"> + + <a href="manage-connections.html" class="btn btn-neutral float-right" title="Managing Connections" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right"></span></a> + + + <a href="initialize-database.html" class="btn btn-neutral" title="Initializing a Database Backend" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left"></span> Previous</a> + + </div> + + + <hr/> + + <div role="contentinfo"> + <p> + + </p> + </div> + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + +</footer> + + </div> + </div> + + </section> + + </div> + + + + + + <script type="text/javascript"> + var DOCUMENTATION_OPTIONS = { + URL_ROOT:'../', + VERSION:'', + LANGUAGE:'None', + COLLAPSE_INDEX:false, + FILE_SUFFIX:'.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt' + }; + </script> + <script type="text/javascript" src="../_static/jquery.js"></script> + <script type="text/javascript" src="../_static/underscore.js"></script> + <script type="text/javascript" src="../_static/doctools.js"></script> + + + + <script type="text/javascript" src="../_static/js/theme.js"></script> + + <script type="text/javascript"> + jQuery(function () { + SphinxRtdTheme.Navigation.enable(true); + }); + </script> + +</body> +</html> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-airflow-site/blob/7d4d7628/howto/run-with-systemd.html ---------------------------------------------------------------------- diff --git a/howto/run-with-systemd.html b/howto/run-with-systemd.html index cb4ae6f..87e85ce 100644 --- a/howto/run-with-systemd.html +++ b/howto/run-with-systemd.html @@ -24,28 +24,19 @@ - - - <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> - - - - - - <link rel="index" title="Index" - href="../genindex.html"/> - <link rel="search" title="Search" href="../search.html"/> - <link rel="top" title="Airflow Documentation" href="../index.html"/> - <link rel="up" title="How-to Guides" href="index.html"/> - <link rel="next" title="Running Airflow with upstart" href="run-with-upstart.html"/> - <link rel="prev" title="Scaling Out with Mesos (community contributed)" href="executor/use-mesos.html"/> + <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" /> + <link rel="stylesheet" href="../_static/pygments.css" type="text/css" /> + <link rel="index" title="Index" href="../genindex.html" /> + <link rel="search" title="Search" href="../search.html" /> + <link rel="next" title="Running Airflow with upstart" href="run-with-upstart.html" /> + <link rel="prev" title="Scaling Out with Mesos (community contributed)" href="executor/use-mesos.html" /> <script src="../_static/js/modernizr.min.js"></script> </head> -<body class="wy-body-for-nav" role="document"> +<body class="wy-body-for-nav"> <div class="wy-grid-for-nav"> @@ -96,6 +87,7 @@ <li class="toctree-l1 current"><a class="reference internal" href="index.html">How-to Guides</a><ul class="current"> <li class="toctree-l2"><a class="reference internal" href="set-config.html">Setting Configuration Options</a></li> <li class="toctree-l2"><a class="reference internal" href="initialize-database.html">Initializing a Database Backend</a></li> +<li class="toctree-l2"><a class="reference internal" href="operator.html">Using Operators</a></li> <li class="toctree-l2"><a class="reference internal" href="manage-connections.html">Managing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="secure-connections.html">Securing Connections</a></li> <li class="toctree-l2"><a class="reference internal" href="write-logs.html">Writing Logs</a></li> @@ -131,7 +123,7 @@ <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap"> - <nav class="wy-nav-top" role="navigation" aria-label="top navigation"> + <nav class="wy-nav-top" aria-label="top navigation"> <i data-toggle="wy-nav-top" class="fa fa-bars"></i> <a href="../index.html">Airflow</a> @@ -139,9 +131,10 @@ </nav> - <div class="wy-nav-content"> + <div class="rst-content"> + @@ -189,22 +182,20 @@ <h1>Running Airflow with systemd<a class="headerlink" href="#running-airflow-with-systemd" title="Permalink to this headline">¶</a></h1> <p>Airflow can integrate with systemd based systems. This makes watching your daemons easy as systemd can take care of restarting a daemon on failure. -In the <code class="docutils literal"><span class="pre">scripts/systemd</span></code> directory you can find unit files that +In the <code class="docutils literal notranslate"><span class="pre">scripts/systemd</span></code> directory you can find unit files that have been tested on Redhat based systems. You can copy those to -<code class="docutils literal"><span class="pre">/usr/lib/systemd/system</span></code>. It is assumed that Airflow will run under -<code class="docutils literal"><span class="pre">airflow:airflow</span></code>. If not (or if you are running on a non Redhat +<code class="docutils literal notranslate"><span class="pre">/usr/lib/systemd/system</span></code>. It is assumed that Airflow will run under +<code class="docutils literal notranslate"><span class="pre">airflow:airflow</span></code>. If not (or if you are running on a non Redhat based system) you probably need to adjust the unit files.</p> -<p>Environment configuration is picked up from <code class="docutils literal"><span class="pre">/etc/sysconfig/airflow</span></code>. -An example file is supplied. Make sure to specify the <code class="docutils literal"><span class="pre">SCHEDULER_RUNS</span></code> +<p>Environment configuration is picked up from <code class="docutils literal notranslate"><span class="pre">/etc/sysconfig/airflow</span></code>. +An example file is supplied. Make sure to specify the <code class="docutils literal notranslate"><span class="pre">SCHEDULER_RUNS</span></code> variable in this file when you run the scheduler. You -can also define here, for example, <code class="docutils literal"><span class="pre">AIRFLOW_HOME</span></code> or <code class="docutils literal"><span class="pre">AIRFLOW_CONFIG</span></code>.</p> +can also define here, for example, <code class="docutils literal notranslate"><span class="pre">AIRFLOW_HOME</span></code> or <code class="docutils literal notranslate"><span class="pre">AIRFLOW_CONFIG</span></code>.</p> </div> </div> - <div class="articleComments"> - - </div> + </div> <footer> @@ -225,7 +216,7 @@ can also define here, for example, <code class="docutils literal"><span class="p </p> </div> - Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. + Built with <a href="http://sphinx-doc.org/";>Sphinx</a> using a <a href="https://github.com/rtfd/sphinx_rtd_theme";>theme</a> provided by <a href="https://readthedocs.org";>Read the Docs</a>. </footer> @@ -244,6 +235,7 @@ can also define here, for example, <code class="docutils literal"><span class="p var DOCUMENTATION_OPTIONS = { URL_ROOT:'../', VERSION:'', + LANGUAGE:'None', COLLAPSE_INDEX:false, FILE_SUFFIX:'.html', HAS_SOURCE: true, @@ -256,19 +248,13 @@ can also define here, for example, <code class="docutils literal"><span class="p - - - <script type="text/javascript" src="../_static/js/theme.js"></script> - + <script type="text/javascript" src="../_static/js/theme.js"></script> - - <script type="text/javascript"> jQuery(function () { - SphinxRtdTheme.StickyNav.enable(); + SphinxRtdTheme.Navigation.enable(true); }); - </script> - + </script> </body> </html> \ No newline at end of file
