This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/airflow-site.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 134ef1b Update asf-site to output generated at bead826
134ef1b is described below
commit 134ef1b71feddbf8a69728db2d59872c96ecf0b8
Author: kaxil <kaxiln...@gmail.com>
AuthorDate: Wed Aug 26 18:02:58 2020 +0000
Update asf-site to output generated at bead826
---
_gen/indexes/en/blog-index.json | 2 +-
_gen/indexes/en/blog-posts.json | 2 +-
blog/airflow-1.10.10/index.html | 35 +++-
.../index.html | 219 ++++++++++++++++-----
blog/airflow-1.10.8-1.10.9/index.html | 35 +++-
blog/airflow-survey/index.html | 4 +-
blog/announcing-new-website/index.html | 4 +-
blog/apache-airflow-for-newcomers/index.html | 8 +-
.../index.html | 4 +-
.../index.html | 4 +-
.../index.html | 4 +-
.../index.html | 4 +-
blog/index.html | 31 +++
blog/index.xml | 170 ++++++++++++++++
.../index.html | 4 +-
blog/tags/release/index.html | 33 +++-
blog/tags/release/index.xml | 172 +++++++++++++++-
index.html | 30 +--
index.xml | 170 ++++++++++++++++
search/index.html | 4 +-
sitemap.xml | 123 ++++++------
tags/index.html | 2 +-
tags/index.xml | 2 +-
use-cases/adobe/index.html | 4 +-
use-cases/big-fish-games/index.html | 4 +-
use-cases/dish/index.html | 4 +-
use-cases/experity/index.html | 4 +-
use-cases/onefootball/index.html | 4 +-
28 files changed, 922 insertions(+), 164 deletions(-)
diff --git a/_gen/indexes/en/blog-index.json b/_gen/indexes/en/blog-index.json
index 1d49821..e2ee9a9 100644
--- a/_gen/indexes/en/blog-index.json
+++ b/_gen/indexes/en/blog-index.json
@@ -1 +1 @@
-{"version":"2.3.8","fields":["title","description","author","content","tags","url"],"fieldVectors":[["title/It's
a \"Breeze\" to develop Apache
Airflow",[0,1.962,1,0.736,2,0.15,3,0.389,4,0.474]],["description/It's a
\"Breeze\" to develop Apache
Airflow",[2,0.158,4,0.323,5,1.337,6,0.631,7,1.337,8,0.792,9,0.631,10,0.631,11,0.631,12,0.502,13,1.337,14,0.502,15,1.337,16,0.086]],["author/It's
a \"Breeze\" to develop Apache Airflow",[11,0.942,17,1.184]],["content/It's a
\"Breeze\" to develop Ap [...]
\ No newline at end of file
+{"version":"2.3.8","fields":["title","description","author","content","tags","url"],"fieldVectors":[["title/It's
a \"Breeze\" to develop Apache
Airflow",[0,2.014,1,0.802,2,0.092,3,0.384,4,0.474]],["description/It's a
\"Breeze\" to develop Apache
Airflow",[2,0.097,4,0.323,5,1.373,6,0.675,7,1.373,8,0.834,9,0.675,10,0.675,11,0.675,12,0.547,13,1.373,14,0.547,15,1.373,16,0.136]],["author/It's
a \"Breeze\" to develop Apache Airflow",[11,1.023,17,1.265]],["content/It's a
\"Breeze\" to develop A [...]
\ No newline at end of file
diff --git a/_gen/indexes/en/blog-posts.json b/_gen/indexes/en/blog-posts.json
index 5b9ded8..cedabb1 100644
--- a/_gen/indexes/en/blog-posts.json
+++ b/_gen/indexes/en/blog-posts.json
@@ -1 +1 @@
-[{"content":"## The story behind the Airflow Breeze tool\nInitially, we
started contributing to this fantastic open-source project [Apache Airflow]
with a team of three which then grew to five. When we kicked it off a year ago,
I realized pretty soon where the biggest bottlenecks and areas for improvement
in terms of productivity were. Even with the help of our client, who provided
us with a “homegrown” development environment it took us literally days to set
it up and learn some basics. [...]
\ No newline at end of file
+[{"content":"## The story behind the Airflow Breeze tool\nInitially, we
started contributing to this fantastic open-source project [Apache Airflow]
with a team of three which then grew to five. When we kicked it off a year ago,
I realized pretty soon where the biggest bottlenecks and areas for improvement
in terms of productivity were. Even with the help of our client, who provided
us with a “homegrown” development environment it took us literally days to set
it up and learn some basics. [...]
\ No newline at end of file
diff --git a/blog/airflow-1.10.10/index.html b/blog/airflow-1.10.10/index.html
index 8ec5b2e..38ec1e9 100644
--- a/blog/airflow-1.10.10/index.html
+++ b/blog/airflow-1.10.10/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-04-09T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow 1.10.10">
<meta itemprop="description" content="We are happy to present Apache Airflow
1.10.10">
<meta itemprop="datePublished" content="2020-04-09T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="1143">
@@ -777,6 +777,37 @@ objc[1873]: +[__NSPlaceholderDate initialize] may have
been in progress in anoth
</div>
+ <span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Tue, Aug 25, 2020</span>
+ </div>
+ <p class="box-event__blogpost--header">Apache Airflow 1.10.12</p>
+ <p class="box-event__blogpost--author">Kaxil Naik</p>
+ <p class="box-event__blogpost--description">We are happy to present
Apache Airflow 1.10.12</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-1.10.12/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read
more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/release/">Release</a>
+
+
+ </div>
<span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Sun, Feb 23, 2020</span>
</div>
<p class="box-event__blogpost--header">Apache Airflow 1.10.8 &
1.10.9</p>
diff --git a/blog/airflow-1.10.8-1.10.9/index.html
b/blog/airflow-1.10.12/index.html
similarity index 84%
copy from blog/airflow-1.10.8-1.10.9/index.html
copy to blog/airflow-1.10.12/index.html
index 9bd0268..559dda4 100644
--- a/blog/airflow-1.10.8-1.10.9/index.html
+++ b/blog/airflow-1.10.12/index.html
@@ -29,21 +29,21 @@
<meta name="msapplication-TileImage" content="/favicons/ms-icon-144x144.png">
<meta name="theme-color" content="#ffffff">
-<title>Apache Airflow 1.10.8 & 1.10.9 | Apache Airflow</title><meta
property="og:title" content="Apache Airflow 1.10.8 & 1.10.9" />
-<meta property="og:description" content="We are happy to present the new
1.10.8 and 1.10.9 releases of Apache Airflow." />
+<title>Apache Airflow 1.10.12 | Apache Airflow</title><meta
property="og:title" content="Apache Airflow 1.10.12" />
+<meta property="og:description" content="We are happy to present Apache
Airflow 1.10.12" />
<meta property="og:type" content="article" />
-<meta property="og:url" content="/blog/airflow-1.10.8-1.10.9/" />
+<meta property="og:url" content="/blog/airflow-1.10.12/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:published_time" content="2020-02-23T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
-<meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
-<meta itemprop="description" content="We are happy to present the new 1.10.8
and 1.10.9 releases of Apache Airflow.">
+<meta property="article:published_time" content="2020-08-25T00:00:00+00:00" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta itemprop="name" content="Apache Airflow 1.10.12">
+<meta itemprop="description" content="We are happy to present Apache Airflow
1.10.12">
-<meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
-<meta itemprop="wordCount" content="437">
+<meta itemprop="datePublished" content="2020-08-25T00:00:00+00:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
+<meta itemprop="wordCount" content="911">
@@ -51,8 +51,8 @@
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="/images/feature-image.png"/>
-<meta name="twitter:title" content="Apache Airflow 1.10.8 & 1.10.9"/>
-<meta name="twitter:description" content="We are happy to present the new
1.10.8 and 1.10.9 releases of Apache Airflow."/>
+<meta name="twitter:title" content="Apache Airflow 1.10.12"/>
+<meta name="twitter:description" content="We are happy to present Apache
Airflow 1.10.12"/>
<script type="application/javascript">
@@ -80,7 +80,7 @@ if (!doNotTrack) {
crossorigin="anonymous"></script>
-<meta name="description" content="We are happy to present the new 1.10.8 and
1.10.9 releases of Apache Airflow." />
+<meta name="description" content="We are happy to present Apache Airflow
1.10.12" />
@@ -439,9 +439,9 @@ if (!doNotTrack) {
</div>
- <span class="bodytext__medium--brownish-grey">Sun, Feb 23, 2020</span>
+ <span class="bodytext__medium--brownish-grey">Tue, Aug 25, 2020</span>
</div>
- <p class="blogpost-content__metadata--title">Apache Airflow 1.10.8 &
1.10.9</p>
+ <p class="blogpost-content__metadata--title">Apache Airflow 1.10.12</p>
<div class="blogpost-content__metadata--author">
<span class="blogpost-content__metadata--author">
Kaxil Naik
@@ -485,55 +485,131 @@ if (!doNotTrack) {
</a>
</div>
- <p class="blogpost-content__metadata--description">We are happy to present
the new 1.10.8 and 1.10.9 releases of Apache Airflow.</p>
+ <p class="blogpost-content__metadata--description">We are happy to present
Apache Airflow 1.10.12</p>
</div>
<div class="markdown-content">
-<p>Airflow 1.10.8 contains 160 commits since 1.10.7 and includes 4 new
features, 42 improvements, 36 bug fixes, and several doc changes.</p>
-
-<p>We released 1.10.9 on the same day as one of the Flask dependencies
(Werkzeug) released 1.0 which broke Airflow 1.10.8.</p>
+<p>Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new
features, 23 improvements, 23 bug fixes,
+and several doc changes.</p>
<p><strong>Details</strong>:</p>
<ul>
-<li><strong>PyPI</strong>: <a
href="https://pypi.org/project/apache-airflow/1.10.9/";
target="_blank">https://pypi.org/project/apache-airflow/1.10.9/</a></li>
-<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.9/";
target="_blank">https://airflow.apache.org/docs/1.10.9/</a></li>
-<li><strong>Changelog (1.10.8)</strong>: <a
href="http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07";
target="_blank">http://airflow.apache.org/docs/1.10.8/changelog.html#airflow-1-10-8-2020-01-07</a></li>
-<li><strong>Changelog (1.10.9)</strong>: <a
href="http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10";
target="_blank">http://airflow.apache.org/docs/1.10.9/changelog.html#airflow-1-10-9-2020-02-10</a></li>
+<li><strong>PyPI</strong>: <a
href="https://pypi.org/project/apache-airflow/1.10.12/";
target="_blank">https://pypi.org/project/apache-airflow/1.10.12/</a></li>
+<li><strong>Docs</strong>: <a href="https://airflow.apache.org/docs/1.10.12/";
target="_blank">https://airflow.apache.org/docs/1.10.12/</a></li>
+<li><strong>Changelog</strong>: <a
href="http://airflow.apache.org/docs/1.10.12/changelog.html";
target="_blank">http://airflow.apache.org/docs/1.10.12/changelog.html</a></li>
</ul>
+<p><strong>Airflow 1.10.11 has breaking changes with respect to
+KubernetesExecutor & KubernetesPodOperator so I recommend users to
directly upgrade to Airflow 1.10.12 instead</strong>.</p>
+
<p>Some of the noteworthy new features (user-facing) are:</p>
<ul>
-<li><a href="https://github.com/apache/airflow/pull/6489"; target="_blank">Add
tags to DAGs and use it for filtering in the UI (RBAC only)</a></li>
-<li><a href="http://airflow.apache.org/docs/1.10.9/executor/debug.html";
target="_blank">New Executor: DebugExecutor for Local debugging from your
IDE</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7281";
target="_blank">Allow passing conf in “Add DAG Run” (Triggered
Dags) view</a></li>
-<li><a href="https://github.com/apache/airflow/pull/7038";
target="_blank">Allow dags to run for future execution dates for manually
triggered DAGs (only if <code>schedule_interval=None</code>)</a></li>
-<li><a href="https://airflow.apache.org/docs/1.10.9/configurations-ref.html";
target="_blank">Dedicated page in documentation for all configs in
airflow.cfg</a></li>
+<li><a href="https://github.com/apache/airflow/pull/8560";
target="_blank">Allow defining custom XCom class</a></li>
+<li><a href="https://github.com/apache/airflow/pull/9645"; target="_blank">Get
Airflow configs with sensitive data from Secret Backends</a></li>
+<li><a href="https://github.com/apache/airflow/pull/10282"; target="_blank">Add
AirflowClusterPolicyViolation support to Airflow local settings</a></li>
+</ul>
+
+<h3 id="allow-defining-custom-xcom-class">Allow defining Custom XCom class</h3>
+
+<p>Until Airflow 1.10.11, the XCom data was only stored in Airflow
Metadatabase. From Airflow 1.10.12, users
+would be able to define custom XCom classes. This will allow users to transfer
larger data between tasks.
+An example here would be to store XCom in S3 or GCS Bucket if the size of data
that needs to be stored is larger
+than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
+
+<p><strong>PR</strong>: <a href="https://github.com/apache/airflow/pull/8560";
target="_blank">https://github.com/apache/airflow/pull/8560</a></p>
+
+<h3 id="get-airflow-configs-with-sensitive-data-from-secret-backends">Get
Airflow configs with sensitive data from Secret Backends</h3>
+
+<p>Users would be able to get the following Airflow configs from Secrets
Backend like Hashicorp Vault:</p>
+
+<ul>
+<li><code>sql_alchemy_conn</code> in [core] section</li>
+<li><code>fernet_key</code> in [core] section</li>
+<li><code>broker_url</code> in [celery] section</li>
+<li><code>flower_basic_auth</code> in [celery] section</li>
+<li><code>result_backend</code> in [celery] section</li>
+<li><code>password</code> in [atlas] section</li>
+<li><code>smtp_password</code> in [smtp] section</li>
+<li><code>bind_password</code> in [ldap] section</li>
+<li><code>git_password</code> in [kubernetes] section</li>
</ul>
-<h3 id="add-tags-to-dags-and-use-it-for-filtering-in-the-ui">Add tags to DAGs
and use it for filtering in the UI</h3>
+<p>Further improving Airflow’s Secret Management story, from Airflow
1.10.12, users don’t need to hardcode
+the <strong>sensitive</strong> config value in airflow.cfg nor then need to
use an Environment variable to set this config.</p>
+
+<p>For example, the metadata database connection string can either be set in
airflow.cfg like this:</p>
+
+<pre><code class="language-ini">[core]
+sql_alchemy_conn_secret = sql_alchemy_conn
+</code></pre>
-<p>In order to filter DAGs (e.g by team), you can add tags in each dag. The
filter is saved in a cookie and can be reset by the reset button.</p>
+<p>This will retrieve config option from the set Secret Backends.</p>
-<p>For example:</p>
+<p>As you can see you just need to add a <code>_secret</code> suffix at the
end of the actual config option
+and the value needs to be the <strong>key</strong> which the Secrets backend
will look for.</p>
-<p>In your Dag file, pass a list of tags you want to add to DAG object:</p>
+<p>Similarly, <code>_secret</code> config options can also be set using a
corresponding environment variable. For example:</p>
-<pre><code class="language-python">dag = DAG(
- dag_id='example_dag_tag',
- schedule_interval='0 0 * * *',
- tags=['example']
-)
+<pre><code>export AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
</code></pre>
-<p><strong>Screenshot</strong>:
-<img src="airflow-dag-tags.png" alt="Add filter by DAG tags" /></p>
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/howto/set-config.html";
target="_blank">http://airflow.apache.org/docs/1.10.12/howto/set-config.html</a></p>
+
+<h3
id="add-airflowclusterpolicyviolation-support-to-airflow-local-settings-py">Add
AirflowClusterPolicyViolation support to airflow_local_settings.py</h3>
+
+<p>Users can use Cluster Policies to apply cluster-wide checks on Airflow
+tasks. You can raise <a
href="http://airflow.apache.org/docs/1.10.12/_api/airflow/exceptions/index.html#airflow.exceptions.AirflowClusterPolicyViolation";
target="_blank">AirflowClusterPolicyViolation</a>
+in a policy or task mutation hook to prevent a DAG from being
+imported or prevent a task from being executed if the task is not compliant
with
+your check.</p>
+
+<p>These checks are intended to help teams using Airflow to protect against
common
+beginner errors that may get past a code reviewer, rather than as technical
+security controls.</p>
-<p><strong>Note</strong>: This feature is only available for the RBAC UI
(enabled using <code>rbac=True</code> in <code>[webserver]</code> section in
your <code>airflow.cfg</code>).</p>
+<p>For example, don’t run tasks without <code>airflow</code> owners:</p>
+
+<pre><code class="language-python">def task_must_have_owners(task):
+ if not task.owner or task.owner.lower() == conf.get('operators',
'default_owner'):
+ raise AirflowClusterPolicyViolation(
+ 'Task must have non-None non-default owner. Current value:
{}'.format(task.owner))
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks";
target="_blank">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
+
+<h3
id="launch-pods-via-yaml-files-when-using-kubernetesexecutor-and-kubernetespodoperator">Launch
Pods via YAML files when using KubernetesExecutor and
KubernetesPodOperator</h3>
+
+<p>As of 1.10.12, users can launch pods via YAML files instead of passing
various configurations.</p>
+
+<p>To allow greater flexibility we have deprecated Airflow’s Pod class
and instead now use classes and
+objects from the official Kubernetes API. The POD class will still work but
raise a deprecation
+warning. This feature involved a pretty extensive rewrite of all of our pod
creation code.</p>
+
+<p>Initially, we were going to hold off on these features until Airflow 2.0.
However, we soon
+realized that exposing these features in 1.10.x is crucial in preparing users
for the 2.0 release to come.</p>
+
+<p>Details: <a href="https://github.com/apache/airflow/pull/6230";
target="_blank">https://github.com/apache/airflow/pull/6230</a> (<a
href="https://github.com/apache/airflow/commit/7aa0f472b57985a952a3e3d0a38f1b2535d93413";
target="_blank">Backport commit</a>)</p>
+
+<h2 id="updating-guide">Updating Guide</h2>
+
+<p>If you are updating Apache Airflow from a previous version to
<code>1.10.12</code>, please take a note of the following:</p>
+
+<ul>
+<li><p>Run <code>airflow upgradedb</code> after <code>pip install -U
apache-airflow==1.10.12</code> as <code>1.10.12</code> contains 1 database
migration.</p></li>
+
+<li><p>As of airflow 1.10.12, using the
<code>airflow.contrib.kubernetes.Pod</code> class in the
<code>pod_mutation_hook</code> is now
+deprecated. Instead we recommend that users treat the pod parameter as a
<code>kubernetes.client.models.V1Pod</code> object.
+This means that users now have access to the full Kubernetes API when
modifying airflow pods for mutating POD.</p></li>
+
+<li><p>Previously, when tasks skipped by SkipMixin (such as
<code>BranchPythonOperator</code>, <code>BaseBranchOperator</code> and
+<code>ShortCircuitOperator</code>) are cleared, they execute. Since 1.10.12,
when such skipped tasks are cleared,
+they will be skipped again by the newly introduced
<code>NotPreviouslySkippedDep</code>.</p></li>
+</ul>
-<h2 id="special-note-deprecations">Special Note / Deprecations</h2>
+<h2 id="special-note">Special Note</h2>
<h3 id="python-2">Python 2</h3>
@@ -544,19 +620,31 @@ Airflow 1.10.* would be the last series to support Python
2.</p>
<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
-<p>Airflow 1.10.9 ships with 2 UIs, the default is non-RBAC Flask-admin based
UI and Flask-appbuilder based UI.</p>
+<p>Airflow 1.10.10 ships with 2 UIs, the default is non-RBAC Flask-admin based
UI and Flask-appbuilder based UI.</p>
-<p>The Flask-AppBuilder (FAB) based UI is allows Role-based Access Control and
has more advanced features compared to
-the legacy Flask-admin based UI. This UI can be enabled by setting
<code>rbac=True</code> in <code>[webserver]</code> section in your
<code>airflow.cfg</code>.</p>
+<p>The Flask-AppBuilder (FAB) based UI allows Role-based Access Control and
has more advanced features compared to
+the legacy Flask-admin based UI. This UI can be enabled by setting
<code>rbac=True</code> in <code>[webserver]</code> section in
+your <code>airflow.cfg</code>.</p>
<p>Flask-admin based UI is deprecated and new features won’t be ported
to it. This UI will still be the default
for 1.10.* series but would no longer be available from Airflow 2.0</p>
+<h3 id="we-have-moved-to-github-issues">We have moved to Github Issues</h3>
+
+<p>The Airflow Project has moved from <a
href="https://issues.apache.org/jira/projects/AIRFLOW/issues";
target="_blank">JIRA</a> to
+<a href="https://github.com/apache/airflow/issues"; target="_blank">Github</a>
for tracking issues.</p>
+
+<p>So if you find any bugs in Airflow 1.10.10 please create a Github Issue for
it.</p>
+
<h2 id="list-of-contributors">List of Contributors</h2>
-<p>According to git shortlog, the following people contributed to the 1.10.8
and 1.10.9 release. Thank you to all contributors!</p>
+<p>According to git shortlog, the following people contributed to the 1.10.10
release. Thank you to all contributors!</p>
-<p>Anita Fronczak, Ash Berlin-Taylor, BasPH, Bharat Kashyap, Bharath Palaksha,
Bhavika Tekwani, Bjorn Olsen, Brian Phillips, Cooper Gillan, Daniel Cohen,
Daniel Imberman, Daniel Standish, Gabriel Eckers, Hossein Torabi, Igor Khrol,
Jacob, Jarek Potiuk, Jay, Jiajie Zhong, Jithin Sukumar, Kamil Breguła, Kaxil
Naik, Kousuke Saruta, Mustafa Gök, Noël Bardelot, Oluwafemi Sule, Pete DeJoy,
QP Hou, Qian Yu, Robin Edwards, Ry Walker, Steven van Rossum, Tomek Urbaszek,
Xinbin Huang, Yuen-Kuei Hsu [...]
+<p>Alexander Sutcliffe, Andy, Aneesh Joseph, Ash Berlin-Taylor, Aviral
Agrawal, BaoshanGu, Beni Ben zikry,
+Daniel Imberman, Daniel Standish, Danylo Baibak, Ephraim Anierobi, Felix
Uellendall, Greg Neiheisel,
+Hartorn, Jacob Ferriero, Jannik F, Jarek Potiuk, Jinhui Zhang, Kamil Breguła,
Kaxil Naik, Kurganov,
+Luis Magana, Max Arrich, Pete DeJoy, Sumit Maheshwari, Tomek Urbaszek, Vicken
Simonian, Vinnie Guimaraes,
+William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314,
retornam, yuqian90</p>
</div>
@@ -614,16 +702,16 @@ for 1.10.* series but would no longer be available from
Airflow 2.0</p>
<div class="pager">
- <a
href="/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/">
+ <a href="/blog/apache-airflow-for-newcomers/">
<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue"
>Previous</button>
</a>
- <a href="/blog/airflow-1.10.10/">
+ <a >
-<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue"
>Next</button>
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue"
disabled>Next</button>
</a>
</div>
@@ -672,6 +760,37 @@ for 1.10.* series but would no longer be available from
Airflow 2.0</p>
</div>
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/release/">Release</a>
+
+
+ </div>
+ <span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Sun, Feb 23, 2020</span>
+ </div>
+ <p class="box-event__blogpost--header">Apache Airflow 1.10.8 &
1.10.9</p>
+ <p class="box-event__blogpost--author">Kaxil Naik</p>
+ <p class="box-event__blogpost--description">We are happy to present
the new 1.10.8 and 1.10.9 releases of Apache Airflow.</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-1.10.8-1.10.9/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read
more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
</div>
</div>
</div>
@@ -684,7 +803,7 @@ for 1.10.* series but would no longer be available from
Airflow 2.0</p>
<div class="base-layout--button">
- <a
href=https://github.com/apache/airflow-site/edit/master/landing-pages/site/content/en/blog/airflow-1.10.8-1.10.9/index.md>
+ <a
href=https://github.com/apache/airflow-site/edit/master/landing-pages/site/content/en/blog/airflow-1.10.12/index.md>
diff --git a/blog/airflow-1.10.8-1.10.9/index.html
b/blog/airflow-1.10.8-1.10.9/index.html
index 9bd0268..bbe84ff 100644
--- a/blog/airflow-1.10.8-1.10.9/index.html
+++ b/blog/airflow-1.10.8-1.10.9/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-02-23T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow 1.10.8 & 1.10.9">
<meta itemprop="description" content="We are happy to present the new 1.10.8
and 1.10.9 releases of Apache Airflow.">
<meta itemprop="datePublished" content="2020-02-23T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="437">
@@ -654,6 +654,37 @@ for 1.10.* series but would no longer be available from
Airflow 2.0</p>
</div>
+ <span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Tue, Aug 25, 2020</span>
+ </div>
+ <p class="box-event__blogpost--header">Apache Airflow 1.10.12</p>
+ <p class="box-event__blogpost--author">Kaxil Naik</p>
+ <p class="box-event__blogpost--description">We are happy to present
Apache Airflow 1.10.12</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-1.10.12/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read
more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/release/">Release</a>
+
+
+ </div>
<span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Thu, Apr 9, 2020</span>
</div>
<p class="box-event__blogpost--header">Apache Airflow 1.10.10</p>
diff --git a/blog/airflow-survey/index.html b/blog/airflow-survey/index.html
index 6e2119e..adc4761 100644
--- a/blog/airflow-survey/index.html
+++ b/blog/airflow-survey/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Airflow Survey 2019">
<meta itemprop="description" content="Receiving and adjusting to our users’
feedback is a must. Let’s see who Airflow users are, how they play with it, and
what they miss.">
<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="1775">
diff --git a/blog/announcing-new-website/index.html
b/blog/announcing-new-website/index.html
index e9c8899..d760ded 100644
--- a/blog/announcing-new-website/index.html
+++ b/blog/announcing-new-website/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-12-11T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="New Airflow website">
<meta itemprop="description" content="We are thrilled about our new website!">
<meta itemprop="datePublished" content="2019-12-11T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="282">
diff --git a/blog/apache-airflow-for-newcomers/index.html
b/blog/apache-airflow-for-newcomers/index.html
index 992ba10..5edf8b5 100644
--- a/blog/apache-airflow-for-newcomers/index.html
+++ b/blog/apache-airflow-for-newcomers/index.html
@@ -37,14 +37,14 @@ Authoring Workflow in Apache Airflow. Airflow makes it easy
to author workflows
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-08-17T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Apache Airflow For Newcomers">
<meta itemprop="description" content="Apache Airflow is a platform to
programmatically author, schedule, and monitor workflows. A workflow is a
sequence of tasks that processes a set of data. You can think of workflow as
the path that describes how tasks go from being undone to done. Scheduling, on
the other hand, is the process of planning, controlling, and optimizing when a
particular task should be done.
Authoring Workflow in Apache Airflow. Airflow makes it easy to author
workflows using python scripts.">
<meta itemprop="datePublished" content="2020-08-17T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="1070">
@@ -687,10 +687,10 @@ and <a href="https://github.com/turbaszek";
target="_blank">Tomek</a> for all the
<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue"
>Previous</button>
</a>
- <a >
+ <a href="/blog/airflow-1.10.12/">
-<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue"
disabled>Next</button>
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue"
>Next</button>
</a>
</div>
diff --git
a/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
b/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
index 2153077..da26086 100644
---
a/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
+++
b/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="ApacheCon Europe 2019 — Thoughts and Insights
by Airflow Committers">
<meta itemprop="description" content="Here come some thoughts by Airflow
committers and contributors from the ApacheCon Europe 2019. Get to know the ASF
community!">
<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="138">
diff --git a/blog/documenting-using-local-development-environments/index.html
b/blog/documenting-using-local-development-environments/index.html
index 0d73b12..244bca2 100644
--- a/blog/documenting-using-local-development-environments/index.html
+++ b/blog/documenting-using-local-development-environments/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Documenting using local development
environment">
<meta itemprop="description" content="The story behind documenting local
development environment of Apache Airflow">
<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="256">
diff --git
a/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
b/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
index 3bad51a..336d58f 100644
---
a/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
+++
b/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/index.html
@@ -37,14 +37,14 @@ About Me I have been writing tech articles on medium as
well as my blog for the
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-12-20T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Experience in Google Season of Docs 2019 with
Apache Airflow">
<meta itemprop="description" content="I came across Google Season of Docs
(GSoD) almost by accident, thanks to my extensive HackerNews and Twitter
addiction. I was familiar with the Google Summer of Code but not with this
program. It turns out it was the inaugural phase. I read the details, and the
process felt a lot like GSoC except that this was about documentation.
About Me I have been writing tech articles on medium as well as my blog for
the past 1.">
<meta itemprop="datePublished" content="2019-12-20T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="1521">
diff --git a/blog/implementing-stable-api-for-apache-airflow/index.html
b/blog/implementing-stable-api-for-apache-airflow/index.html
index 5271b51..d57eae6 100644
--- a/blog/implementing-stable-api-for-apache-airflow/index.html
+++ b/blog/implementing-stable-api-for-apache-airflow/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2020-07-19T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Implementing Stable API for Apache Airflow">
<meta itemprop="description" content="An Outreachy intern's progress
report on contributing to Apache Airflow REST API.">
<meta itemprop="datePublished" content="2020-07-19T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="703">
diff --git a/blog/index.html b/blog/index.html
index 3848457..286238b 100644
--- a/blog/index.html
+++ b/blog/index.html
@@ -484,6 +484,37 @@ if (!doNotTrack) {
<div class="tags-container">
+ <a class="tag" href="/blog/tags/release/">Release</a>
+
+
+ </div>
+ <span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Tue, Aug 25, 2020</span>
+ </div>
+ <p class="box-event__blogpost--header">Apache Airflow 1.10.12</p>
+ <p class="box-event__blogpost--author">Kaxil Naik</p>
+ <p class="box-event__blogpost--description">We are happy to present
Apache Airflow 1.10.12</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-1.10.12/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read
more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
<a class="tag"
href="/blog/tags/community/">Community</a>
diff --git a/blog/index.xml b/blog/index.xml
index 82338ad..4202ffb 100644
--- a/blog/index.xml
+++ b/blog/index.xml
@@ -14,6 +14,176 @@
<item>
+ <title>Blog: Apache Airflow 1.10.12</title>
+ <link>/blog/airflow-1.10.12/</link>
+ <pubDate>Tue, 25 Aug 2020 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-1.10.12/</guid>
+ <description>
+
+
+
+
+<p>Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new
features, 23 improvements, 23 bug fixes,
+and several doc changes.</p>
+
+<p><strong>Details</strong>:</p>
+
+<ul>
+<li><strong>PyPI</strong>: <a
href="https://pypi.org/project/apache-airflow/1.10.12/"
target="_blank">https://pypi.org/project/apache-airflow/1.10.12/</a></li>
+<li><strong>Docs</strong>: <a
href="https://airflow.apache.org/docs/1.10.12/"
target="_blank">https://airflow.apache.org/docs/1.10.12/</a></li>
+<li><strong>Changelog</strong>: <a
href="http://airflow.apache.org/docs/1.10.12/changelog.html"
target="_blank">http://airflow.apache.org/docs/1.10.12/changelog.html</a></li>
+</ul>
+
+<p><strong>Airflow 1.10.11 has breaking changes with respect to
+KubernetesExecutor &amp; KubernetesPodOperator so I recommend users to
directly upgrade to Airflow 1.10.12 instead</strong>.</p>
+
+<p>Some of the noteworthy new features (user-facing) are:</p>
+
+<ul>
+<li><a href="https://github.com/apache/airflow/pull/8560"
target="_blank">Allow defining custom XCom class</a></li>
+<li><a href="https://github.com/apache/airflow/pull/9645"
target="_blank">Get Airflow configs with sensitive data from Secret
Backends</a></li>
+<li><a href="https://github.com/apache/airflow/pull/10282"
target="_blank">Add AirflowClusterPolicyViolation support to Airflow
local settings</a></li>
+</ul>
+
+<h3 id="allow-defining-custom-xcom-class">Allow defining Custom
XCom class</h3>
+
+<p>Until Airflow 1.10.11, the XCom data was only stored in Airflow
Metadatabase. From Airflow 1.10.12, users
+would be able to define custom XCom classes. This will allow users to transfer
larger data between tasks.
+An example here would be to store XCom in S3 or GCS Bucket if the size of data
that needs to be stored is larger
+than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
+
+<p><strong>PR</strong>: <a
href="https://github.com/apache/airflow/pull/8560"
target="_blank">https://github.com/apache/airflow/pull/8560</a></p>
+
+<h3
id="get-airflow-configs-with-sensitive-data-from-secret-backends">Get
Airflow configs with sensitive data from Secret Backends</h3>
+
+<p>Users would be able to get the following Airflow configs from Secrets
Backend like Hashicorp Vault:</p>
+
+<ul>
+<li><code>sql_alchemy_conn</code> in [core]
section</li>
+<li><code>fernet_key</code> in [core] section</li>
+<li><code>broker_url</code> in [celery] section</li>
+<li><code>flower_basic_auth</code> in [celery]
section</li>
+<li><code>result_backend</code> in [celery]
section</li>
+<li><code>password</code> in [atlas] section</li>
+<li><code>smtp_password</code> in [smtp] section</li>
+<li><code>bind_password</code> in [ldap] section</li>
+<li><code>git_password</code> in [kubernetes]
section</li>
+</ul>
+
+<p>Further improving Airflow&rsquo;s Secret Management story, from
Airflow 1.10.12, users don&rsquo;t need to hardcode
+the <strong>sensitive</strong> config value in airflow.cfg nor
then need to use an Environment variable to set this config.</p>
+
+<p>For example, the metadata database connection string can either be
set in airflow.cfg like this:</p>
+
+<pre><code class="language-ini">[core]
+sql_alchemy_conn_secret = sql_alchemy_conn
+</code></pre>
+
+<p>This will retrieve config option from the set Secret
Backends.</p>
+
+<p>As you can see you just need to add a
<code>_secret</code> suffix at the end of the actual config option
+and the value needs to be the <strong>key</strong> which the
Secrets backend will look for.</p>
+
+<p>Similarly, <code>_secret</code> config options can also
be set using a corresponding environment variable. For example:</p>
+
+<pre><code>export
AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/howto/set-config.html"
target="_blank">http://airflow.apache.org/docs/1.10.12/howto/set-config.html</a></p>
+
+<h3
id="add-airflowclusterpolicyviolation-support-to-airflow-local-settings-py">Add
AirflowClusterPolicyViolation support to airflow_local_settings.py</h3>
+
+<p>Users can use Cluster Policies to apply cluster-wide checks on Airflow
+tasks. You can raise <a
href="http://airflow.apache.org/docs/1.10.12/_api/airflow/exceptions/index.html#airflow.exceptions.AirflowClusterPolicyViolation"
target="_blank">AirflowClusterPolicyViolation</a>
+in a policy or task mutation hook to prevent a DAG from being
+imported or prevent a task from being executed if the task is not compliant
with
+your check.</p>
+
+<p>These checks are intended to help teams using Airflow to protect
against common
+beginner errors that may get past a code reviewer, rather than as technical
+security controls.</p>
+
+<p>For example, don&rsquo;t run tasks without
<code>airflow</code> owners:</p>
+
+<pre><code class="language-python">def
task_must_have_owners(task):
+ if not task.owner or task.owner.lower() == conf.get('operators',
'default_owner'):
+ raise AirflowClusterPolicyViolation(
+ 'Task must have non-None non-default owner. Current value:
{}'.format(task.owner))
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks"
target="_blank">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
+
+<h3
id="launch-pods-via-yaml-files-when-using-kubernetesexecutor-and-kubernetespodoperator">Launch
Pods via YAML files when using KubernetesExecutor and
KubernetesPodOperator</h3>
+
+<p>As of 1.10.12, users can launch pods via YAML files instead of
passing various configurations.</p>
+
+<p>To allow greater flexibility we have deprecated Airflow&rsquo;s
Pod class and instead now use classes and
+objects from the official Kubernetes API. The POD class will still work but
raise a deprecation
+warning. This feature involved a pretty extensive rewrite of all of our pod
creation code.</p>
+
+<p>Initially, we were going to hold off on these features until Airflow
2.0. However, we soon
+realized that exposing these features in 1.10.x is crucial in preparing users
for the 2.0 release to come.</p>
+
+<p>Details: <a
href="https://github.com/apache/airflow/pull/6230"
target="_blank">https://github.com/apache/airflow/pull/6230</a>
(<a
href="https://github.com/apache/airflow/commit/7aa0f472b57985a952a3e3d0a38f1b2535d93413"
target="_blank">Backport commit</a>)</p>
+
+<h2 id="updating-guide">Updating Guide</h2>
+
+<p>If you are updating Apache Airflow from a previous version to
<code>1.10.12</code>, please take a note of the following:</p>
+
+<ul>
+<li><p>Run <code>airflow upgradedb</code> after
<code>pip install -U apache-airflow==1.10.12</code> as
<code>1.10.12</code> contains 1 database
migration.</p></li>
+
+<li><p>As of airflow 1.10.12, using the
<code>airflow.contrib.kubernetes.Pod</code> class in the
<code>pod_mutation_hook</code> is now
+deprecated. Instead we recommend that users treat the pod parameter as a
<code>kubernetes.client.models.V1Pod</code> object.
+This means that users now have access to the full Kubernetes API when
modifying airflow pods for mutating POD.</p></li>
+
+<li><p>Previously, when tasks skipped by SkipMixin (such as
<code>BranchPythonOperator</code>,
<code>BaseBranchOperator</code> and
+<code>ShortCircuitOperator</code>) are cleared, they execute.
Since 1.10.12, when such skipped tasks are cleared,
+they will be skipped again by the newly introduced
<code>NotPreviouslySkippedDep</code>.</p></li>
+</ul>
+
+<h2 id="special-note">Special Note</h2>
+
+<h3 id="python-2">Python 2</h3>
+
+<p>Python 2 has reached end of its life on Jan 2020. Airflow Master no
longer supports Python 2.
+Airflow 1.10.* would be the last series to support Python 2.</p>
+
+<p>We strongly recommend users to use Python &gt;= 3.6</p>
+
+<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
+
+<p>Airflow 1.10.10 ships with 2 UIs, the default is non-RBAC Flask-admin
based UI and Flask-appbuilder based UI.</p>
+
+<p>The Flask-AppBuilder (FAB) based UI allows Role-based Access Control
and has more advanced features compared to
+the legacy Flask-admin based UI. This UI can be enabled by setting
<code>rbac=True</code> in <code>[webserver]</code>
section in
+your <code>airflow.cfg</code>.</p>
+
+<p>Flask-admin based UI is deprecated and new features won&rsquo;t
be ported to it. This UI will still be the default
+for 1.10.* series but would no longer be available from Airflow 2.0</p>
+
+<h3 id="we-have-moved-to-github-issues">We have moved to Github
Issues</h3>
+
+<p>The Airflow Project has moved from <a
href="https://issues.apache.org/jira/projects/AIRFLOW/issues"
target="_blank">JIRA</a> to
+<a href="https://github.com/apache/airflow/issues"
target="_blank">Github</a> for tracking issues.</p>
+
+<p>So if you find any bugs in Airflow 1.10.10 please create a Github
Issue for it.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following people contributed to the
1.10.10 release. Thank you to all contributors!</p>
+
+<p>Alexander Sutcliffe, Andy, Aneesh Joseph, Ash Berlin-Taylor, Aviral
Agrawal, BaoshanGu, Beni Ben zikry,
+Daniel Imberman, Daniel Standish, Danylo Baibak, Ephraim Anierobi, Felix
Uellendall, Greg Neiheisel,
+Hartorn, Jacob Ferriero, Jannik F, Jarek Potiuk, Jinhui Zhang, Kamil Breguła,
Kaxil Naik, Kurganov,
+Luis Magana, Max Arrich, Pete DeJoy, Sumit Maheshwari, Tomek Urbaszek, Vicken
Simonian, Vinnie Guimaraes,
+William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314,
retornam, yuqian90</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Apache Airflow For Newcomers</title>
<link>/blog/apache-airflow-for-newcomers/</link>
<pubDate>Mon, 17 Aug 2020 00:00:00 +0000</pubDate>
diff --git a/blog/its-a-breeze-to-develop-apache-airflow/index.html
b/blog/its-a-breeze-to-develop-apache-airflow/index.html
index 9b3b9ac..a2b8042 100644
--- a/blog/its-a-breeze-to-develop-apache-airflow/index.html
+++ b/blog/its-a-breeze-to-develop-apache-airflow/index.html
@@ -36,13 +36,13 @@
<meta property="og:image" content="/images/feature-image.png" />
<meta property="article:published_time" content="2019-11-22T00:00:00+00:00" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="It's a "Breeze" to develop Apache
Airflow">
<meta itemprop="description" content="A Principal Software Engineer's
journey to developer productivity. Learn how Jarek and his team speeded up and
simplified Airflow development for the community.">
<meta itemprop="datePublished" content="2019-11-22T00:00:00+00:00" />
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="127">
diff --git a/blog/tags/release/index.html b/blog/tags/release/index.html
index 4790c39..51c5058 100644
--- a/blog/tags/release/index.html
+++ b/blog/tags/release/index.html
@@ -37,7 +37,7 @@
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:updated_time" content="2020-04-09T00:00:00+00:00" /><meta
property="og:site_name" content="Apache Airflow" />
+<meta property="og:updated_time" content="2020-08-25T00:00:00+00:00" /><meta
property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="release">
<meta itemprop="description" content="Platform created by the community to
programmatically author, schedule and monitor workflows.">
@@ -497,6 +497,37 @@ if (!doNotTrack) {
</div>
+ <span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Tue, Aug 25, 2020</span>
+ </div>
+ <p class="box-event__blogpost--header">Apache Airflow 1.10.12</p>
+ <p class="box-event__blogpost--author">Kaxil Naik</p>
+ <p class="box-event__blogpost--description">We are happy to present
Apache Airflow 1.10.12</p>
+ <div class="mt-auto">
+ <a href="/blog/airflow-1.10.12/">
+
+
+<button class="btn-hollow btn-blue bodytext__medium--cerulean-blue" >Read
more</button>
+
+ </a>
+ </div>
+ </div>
+</div>
+
+ </div>
+
+ <div class="list-item list-item--wide">
+
+
+<div class="card">
+ <div class="box-event__blogpost">
+ <div class="box-event__blogpost--metadata">
+ <div class="tags-container">
+
+
+ <a class="tag" href="/blog/tags/release/">Release</a>
+
+
+ </div>
<span class="bodytext__medium--brownish-grey
box-event__blogpost--date">Thu, Apr 9, 2020</span>
</div>
<p class="box-event__blogpost--header">Apache Airflow 1.10.10</p>
diff --git a/blog/tags/release/index.xml b/blog/tags/release/index.xml
index e9304b2..5613a7b 100644
--- a/blog/tags/release/index.xml
+++ b/blog/tags/release/index.xml
@@ -4,7 +4,7 @@
<link>/blog/tags/release/</link>
<description>Recent content in release on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
- <lastBuildDate>Thu, 09 Apr 2020 00:00:00 +0000</lastBuildDate>
+ <lastBuildDate>Tue, 25 Aug 2020 00:00:00 +0000</lastBuildDate>
<atom:link href="/blog/tags/release/index.xml" rel="self"
type="application/rss+xml" />
@@ -15,6 +15,176 @@
<item>
+ <title>Blog: Apache Airflow 1.10.12</title>
+ <link>/blog/airflow-1.10.12/</link>
+ <pubDate>Tue, 25 Aug 2020 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-1.10.12/</guid>
+ <description>
+
+
+
+
+<p>Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new
features, 23 improvements, 23 bug fixes,
+and several doc changes.</p>
+
+<p><strong>Details</strong>:</p>
+
+<ul>
+<li><strong>PyPI</strong>: <a
href="https://pypi.org/project/apache-airflow/1.10.12/"
target="_blank">https://pypi.org/project/apache-airflow/1.10.12/</a></li>
+<li><strong>Docs</strong>: <a
href="https://airflow.apache.org/docs/1.10.12/"
target="_blank">https://airflow.apache.org/docs/1.10.12/</a></li>
+<li><strong>Changelog</strong>: <a
href="http://airflow.apache.org/docs/1.10.12/changelog.html"
target="_blank">http://airflow.apache.org/docs/1.10.12/changelog.html</a></li>
+</ul>
+
+<p><strong>Airflow 1.10.11 has breaking changes with respect to
+KubernetesExecutor &amp; KubernetesPodOperator so I recommend users to
directly upgrade to Airflow 1.10.12 instead</strong>.</p>
+
+<p>Some of the noteworthy new features (user-facing) are:</p>
+
+<ul>
+<li><a href="https://github.com/apache/airflow/pull/8560"
target="_blank">Allow defining custom XCom class</a></li>
+<li><a href="https://github.com/apache/airflow/pull/9645"
target="_blank">Get Airflow configs with sensitive data from Secret
Backends</a></li>
+<li><a href="https://github.com/apache/airflow/pull/10282"
target="_blank">Add AirflowClusterPolicyViolation support to Airflow
local settings</a></li>
+</ul>
+
+<h3 id="allow-defining-custom-xcom-class">Allow defining Custom
XCom class</h3>
+
+<p>Until Airflow 1.10.11, the XCom data was only stored in Airflow
Metadatabase. From Airflow 1.10.12, users
+would be able to define custom XCom classes. This will allow users to transfer
larger data between tasks.
+An example here would be to store XCom in S3 or GCS Bucket if the size of data
that needs to be stored is larger
+than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
+
+<p><strong>PR</strong>: <a
href="https://github.com/apache/airflow/pull/8560"
target="_blank">https://github.com/apache/airflow/pull/8560</a></p>
+
+<h3
id="get-airflow-configs-with-sensitive-data-from-secret-backends">Get
Airflow configs with sensitive data from Secret Backends</h3>
+
+<p>Users would be able to get the following Airflow configs from Secrets
Backend like Hashicorp Vault:</p>
+
+<ul>
+<li><code>sql_alchemy_conn</code> in [core]
section</li>
+<li><code>fernet_key</code> in [core] section</li>
+<li><code>broker_url</code> in [celery] section</li>
+<li><code>flower_basic_auth</code> in [celery]
section</li>
+<li><code>result_backend</code> in [celery]
section</li>
+<li><code>password</code> in [atlas] section</li>
+<li><code>smtp_password</code> in [smtp] section</li>
+<li><code>bind_password</code> in [ldap] section</li>
+<li><code>git_password</code> in [kubernetes]
section</li>
+</ul>
+
+<p>Further improving Airflow&rsquo;s Secret Management story, from
Airflow 1.10.12, users don&rsquo;t need to hardcode
+the <strong>sensitive</strong> config value in airflow.cfg nor
then need to use an Environment variable to set this config.</p>
+
+<p>For example, the metadata database connection string can either be
set in airflow.cfg like this:</p>
+
+<pre><code class="language-ini">[core]
+sql_alchemy_conn_secret = sql_alchemy_conn
+</code></pre>
+
+<p>This will retrieve config option from the set Secret
Backends.</p>
+
+<p>As you can see you just need to add a
<code>_secret</code> suffix at the end of the actual config option
+and the value needs to be the <strong>key</strong> which the
Secrets backend will look for.</p>
+
+<p>Similarly, <code>_secret</code> config options can also
be set using a corresponding environment variable. For example:</p>
+
+<pre><code>export
AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/howto/set-config.html"
target="_blank">http://airflow.apache.org/docs/1.10.12/howto/set-config.html</a></p>
+
+<h3
id="add-airflowclusterpolicyviolation-support-to-airflow-local-settings-py">Add
AirflowClusterPolicyViolation support to airflow_local_settings.py</h3>
+
+<p>Users can use Cluster Policies to apply cluster-wide checks on Airflow
+tasks. You can raise <a
href="http://airflow.apache.org/docs/1.10.12/_api/airflow/exceptions/index.html#airflow.exceptions.AirflowClusterPolicyViolation"
target="_blank">AirflowClusterPolicyViolation</a>
+in a policy or task mutation hook to prevent a DAG from being
+imported or prevent a task from being executed if the task is not compliant
with
+your check.</p>
+
+<p>These checks are intended to help teams using Airflow to protect
against common
+beginner errors that may get past a code reviewer, rather than as technical
+security controls.</p>
+
+<p>For example, don&rsquo;t run tasks without
<code>airflow</code> owners:</p>
+
+<pre><code class="language-python">def
task_must_have_owners(task):
+ if not task.owner or task.owner.lower() == conf.get('operators',
'default_owner'):
+ raise AirflowClusterPolicyViolation(
+ 'Task must have non-None non-default owner. Current value:
{}'.format(task.owner))
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks"
target="_blank">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
+
+<h3
id="launch-pods-via-yaml-files-when-using-kubernetesexecutor-and-kubernetespodoperator">Launch
Pods via YAML files when using KubernetesExecutor and
KubernetesPodOperator</h3>
+
+<p>As of 1.10.12, users can launch pods via YAML files instead of
passing various configurations.</p>
+
+<p>To allow greater flexibility we have deprecated Airflow&rsquo;s
Pod class and instead now use classes and
+objects from the official Kubernetes API. The POD class will still work but
raise a deprecation
+warning. This feature involved a pretty extensive rewrite of all of our pod
creation code.</p>
+
+<p>Initially, we were going to hold off on these features until Airflow
2.0. However, we soon
+realized that exposing these features in 1.10.x is crucial in preparing users
for the 2.0 release to come.</p>
+
+<p>Details: <a
href="https://github.com/apache/airflow/pull/6230"
target="_blank">https://github.com/apache/airflow/pull/6230</a>
(<a
href="https://github.com/apache/airflow/commit/7aa0f472b57985a952a3e3d0a38f1b2535d93413"
target="_blank">Backport commit</a>)</p>
+
+<h2 id="updating-guide">Updating Guide</h2>
+
+<p>If you are updating Apache Airflow from a previous version to
<code>1.10.12</code>, please take a note of the following:</p>
+
+<ul>
+<li><p>Run <code>airflow upgradedb</code> after
<code>pip install -U apache-airflow==1.10.12</code> as
<code>1.10.12</code> contains 1 database
migration.</p></li>
+
+<li><p>As of airflow 1.10.12, using the
<code>airflow.contrib.kubernetes.Pod</code> class in the
<code>pod_mutation_hook</code> is now
+deprecated. Instead we recommend that users treat the pod parameter as a
<code>kubernetes.client.models.V1Pod</code> object.
+This means that users now have access to the full Kubernetes API when
modifying airflow pods for mutating POD.</p></li>
+
+<li><p>Previously, when tasks skipped by SkipMixin (such as
<code>BranchPythonOperator</code>,
<code>BaseBranchOperator</code> and
+<code>ShortCircuitOperator</code>) are cleared, they execute.
Since 1.10.12, when such skipped tasks are cleared,
+they will be skipped again by the newly introduced
<code>NotPreviouslySkippedDep</code>.</p></li>
+</ul>
+
+<h2 id="special-note">Special Note</h2>
+
+<h3 id="python-2">Python 2</h3>
+
+<p>Python 2 has reached end of its life on Jan 2020. Airflow Master no
longer supports Python 2.
+Airflow 1.10.* would be the last series to support Python 2.</p>
+
+<p>We strongly recommend users to use Python &gt;= 3.6</p>
+
+<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
+
+<p>Airflow 1.10.10 ships with 2 UIs, the default is non-RBAC Flask-admin
based UI and Flask-appbuilder based UI.</p>
+
+<p>The Flask-AppBuilder (FAB) based UI allows Role-based Access Control
and has more advanced features compared to
+the legacy Flask-admin based UI. This UI can be enabled by setting
<code>rbac=True</code> in <code>[webserver]</code>
section in
+your <code>airflow.cfg</code>.</p>
+
+<p>Flask-admin based UI is deprecated and new features won&rsquo;t
be ported to it. This UI will still be the default
+for 1.10.* series but would no longer be available from Airflow 2.0</p>
+
+<h3 id="we-have-moved-to-github-issues">We have moved to Github
Issues</h3>
+
+<p>The Airflow Project has moved from <a
href="https://issues.apache.org/jira/projects/AIRFLOW/issues"
target="_blank">JIRA</a> to
+<a href="https://github.com/apache/airflow/issues"
target="_blank">Github</a> for tracking issues.</p>
+
+<p>So if you find any bugs in Airflow 1.10.10 please create a Github
Issue for it.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following people contributed to the
1.10.10 release. Thank you to all contributors!</p>
+
+<p>Alexander Sutcliffe, Andy, Aneesh Joseph, Ash Berlin-Taylor, Aviral
Agrawal, BaoshanGu, Beni Ben zikry,
+Daniel Imberman, Daniel Standish, Danylo Baibak, Ephraim Anierobi, Felix
Uellendall, Greg Neiheisel,
+Hartorn, Jacob Ferriero, Jannik F, Jarek Potiuk, Jinhui Zhang, Kamil Breguła,
Kaxil Naik, Kurganov,
+Luis Magana, Max Arrich, Pete DeJoy, Sumit Maheshwari, Tomek Urbaszek, Vicken
Simonian, Vinnie Guimaraes,
+William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314,
retornam, yuqian90</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Apache Airflow 1.10.10</title>
<link>/blog/airflow-1.10.10/</link>
<pubDate>Thu, 09 Apr 2020 00:00:00 +0000</pubDate>
diff --git a/index.html b/index.html
index 824f8ee..99a16d1 100644
--- a/index.html
+++ b/index.html
@@ -1226,12 +1226,12 @@ if (!doNotTrack) {
<div id="integrations-container" class="list-items">
- <a class="list-item"
href="/docs/stable/integration.html#gcp-google-cloud-platform">
+ <a class="list-item"
href="/docs/stable/integration.html#azure-microsoft-azure">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Cloud Storage (GCS)</span>
+ <span class="box-event__integration--name">Azure Files</span>
</div>
</div>
@@ -1239,12 +1239,12 @@ if (!doNotTrack) {
- <a class="list-item"
href="/docs/stable/integration.html#gcp-google-cloud-platform">
+ <a class="list-item"
href="/docs/stable/integration.html#service-integrations">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Cloud Speech-to-Text</span>
+ <span class="box-event__integration--name">Databricks</span>
</div>
</div>
@@ -1252,12 +1252,12 @@ if (!doNotTrack) {
- <a class="list-item"
href="/docs/stable/integration.html#gcp-google-cloud-platform">
+ <a class="list-item"
href="/docs/stable/integration.html#asf-apache-software-foundation">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Datastore</span>
+ <span class="box-event__integration--name">Apache Sqoop</span>
</div>
</div>
@@ -1265,12 +1265,12 @@ if (!doNotTrack) {
- <a class="list-item"
href="/docs/stable/integration.html#aws-amazon-web-services">
+ <a class="list-item"
href="/docs/stable/integration.html#software-integrations">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">AWS Glue Catalog</span>
+ <span class="box-event__integration--name">Samba</span>
</div>
</div>
@@ -1278,12 +1278,12 @@ if (!doNotTrack) {
- <a class="list-item"
href="/docs/stable/integration.html#gcp-google-cloud-platform">
+ <a class="list-item"
href="/docs/stable/integration.html#azure-microsoft-azure">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Cloud SQL</span>
+ <span class="box-event__integration--name">Azure Data Lake
Storage</span>
</div>
</div>
@@ -1291,12 +1291,12 @@ if (!doNotTrack) {
- <a class="list-item"
href="/docs/stable/integration.html#azure-microsoft-azure">
+ <a class="list-item"
href="/docs/stable/integration.html#software-integrations">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Azure Data Lake
Storage</span>
+ <span class="box-event__integration--name">Celery</span>
</div>
</div>
@@ -1304,12 +1304,12 @@ if (!doNotTrack) {
- <a class="list-item"
href="/docs/stable/integration.html#service-integrations">
+ <a class="list-item"
href="/docs/stable/integration.html#gcp-google-cloud-platform">
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Salesforce</span>
+ <span class="box-event__integration--name">Cloud Storage (GCS)</span>
</div>
</div>
@@ -1322,7 +1322,7 @@ if (!doNotTrack) {
<div class="card">
<div class="box-event box-event__integration">
- <span class="box-event__integration--name">Bigtable</span>
+ <span class="box-event__integration--name">Kubernetes Engine</span>
</div>
</div>
diff --git a/index.xml b/index.xml
index 9bef47a..23ed03a 100644
--- a/index.xml
+++ b/index.xml
@@ -13,6 +13,176 @@
<item>
+ <title>Blog: Apache Airflow 1.10.12</title>
+ <link>/blog/airflow-1.10.12/</link>
+ <pubDate>Tue, 25 Aug 2020 00:00:00 +0000</pubDate>
+
+ <guid>/blog/airflow-1.10.12/</guid>
+ <description>
+
+
+
+
+<p>Airflow 1.10.12 contains 113 commits since 1.10.11 and includes 5 new
features, 23 improvements, 23 bug fixes,
+and several doc changes.</p>
+
+<p><strong>Details</strong>:</p>
+
+<ul>
+<li><strong>PyPI</strong>: <a
href="https://pypi.org/project/apache-airflow/1.10.12/"
target="_blank">https://pypi.org/project/apache-airflow/1.10.12/</a></li>
+<li><strong>Docs</strong>: <a
href="https://airflow.apache.org/docs/1.10.12/"
target="_blank">https://airflow.apache.org/docs/1.10.12/</a></li>
+<li><strong>Changelog</strong>: <a
href="http://airflow.apache.org/docs/1.10.12/changelog.html"
target="_blank">http://airflow.apache.org/docs/1.10.12/changelog.html</a></li>
+</ul>
+
+<p><strong>Airflow 1.10.11 has breaking changes with respect to
+KubernetesExecutor &amp; KubernetesPodOperator so I recommend users to
directly upgrade to Airflow 1.10.12 instead</strong>.</p>
+
+<p>Some of the noteworthy new features (user-facing) are:</p>
+
+<ul>
+<li><a href="https://github.com/apache/airflow/pull/8560"
target="_blank">Allow defining custom XCom class</a></li>
+<li><a href="https://github.com/apache/airflow/pull/9645"
target="_blank">Get Airflow configs with sensitive data from Secret
Backends</a></li>
+<li><a href="https://github.com/apache/airflow/pull/10282"
target="_blank">Add AirflowClusterPolicyViolation support to Airflow
local settings</a></li>
+</ul>
+
+<h3 id="allow-defining-custom-xcom-class">Allow defining Custom
XCom class</h3>
+
+<p>Until Airflow 1.10.11, the XCom data was only stored in Airflow
Metadatabase. From Airflow 1.10.12, users
+would be able to define custom XCom classes. This will allow users to transfer
larger data between tasks.
+An example here would be to store XCom in S3 or GCS Bucket if the size of data
that needs to be stored is larger
+than <code>XCom.MAX_XCOM_SIZE</code> (48 KB).</p>
+
+<p><strong>PR</strong>: <a
href="https://github.com/apache/airflow/pull/8560"
target="_blank">https://github.com/apache/airflow/pull/8560</a></p>
+
+<h3
id="get-airflow-configs-with-sensitive-data-from-secret-backends">Get
Airflow configs with sensitive data from Secret Backends</h3>
+
+<p>Users would be able to get the following Airflow configs from Secrets
Backend like Hashicorp Vault:</p>
+
+<ul>
+<li><code>sql_alchemy_conn</code> in [core]
section</li>
+<li><code>fernet_key</code> in [core] section</li>
+<li><code>broker_url</code> in [celery] section</li>
+<li><code>flower_basic_auth</code> in [celery]
section</li>
+<li><code>result_backend</code> in [celery]
section</li>
+<li><code>password</code> in [atlas] section</li>
+<li><code>smtp_password</code> in [smtp] section</li>
+<li><code>bind_password</code> in [ldap] section</li>
+<li><code>git_password</code> in [kubernetes]
section</li>
+</ul>
+
+<p>Further improving Airflow&rsquo;s Secret Management story, from
Airflow 1.10.12, users don&rsquo;t need to hardcode
+the <strong>sensitive</strong> config value in airflow.cfg nor
then need to use an Environment variable to set this config.</p>
+
+<p>For example, the metadata database connection string can either be
set in airflow.cfg like this:</p>
+
+<pre><code class="language-ini">[core]
+sql_alchemy_conn_secret = sql_alchemy_conn
+</code></pre>
+
+<p>This will retrieve config option from the set Secret
Backends.</p>
+
+<p>As you can see you just need to add a
<code>_secret</code> suffix at the end of the actual config option
+and the value needs to be the <strong>key</strong> which the
Secrets backend will look for.</p>
+
+<p>Similarly, <code>_secret</code> config options can also
be set using a corresponding environment variable. For example:</p>
+
+<pre><code>export
AIRFLOW__CORE__SQL_ALCHEMY_CONN_SECRET=sql_alchemy_conn
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/howto/set-config.html"
target="_blank">http://airflow.apache.org/docs/1.10.12/howto/set-config.html</a></p>
+
+<h3
id="add-airflowclusterpolicyviolation-support-to-airflow-local-settings-py">Add
AirflowClusterPolicyViolation support to airflow_local_settings.py</h3>
+
+<p>Users can use Cluster Policies to apply cluster-wide checks on Airflow
+tasks. You can raise <a
href="http://airflow.apache.org/docs/1.10.12/_api/airflow/exceptions/index.html#airflow.exceptions.AirflowClusterPolicyViolation"
target="_blank">AirflowClusterPolicyViolation</a>
+in a policy or task mutation hook to prevent a DAG from being
+imported or prevent a task from being executed if the task is not compliant
with
+your check.</p>
+
+<p>These checks are intended to help teams using Airflow to protect
against common
+beginner errors that may get past a code reviewer, rather than as technical
+security controls.</p>
+
+<p>For example, don&rsquo;t run tasks without
<code>airflow</code> owners:</p>
+
+<pre><code class="language-python">def
task_must_have_owners(task):
+ if not task.owner or task.owner.lower() == conf.get('operators',
'default_owner'):
+ raise AirflowClusterPolicyViolation(
+ 'Task must have non-None non-default owner. Current value:
{}'.format(task.owner))
+</code></pre>
+
+<p>More details: <a
href="http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks"
target="_blank">http://airflow.apache.org/docs/1.10.12/concepts.html#cluster-policies-for-custom-task-checks</a></p>
+
+<h3
id="launch-pods-via-yaml-files-when-using-kubernetesexecutor-and-kubernetespodoperator">Launch
Pods via YAML files when using KubernetesExecutor and
KubernetesPodOperator</h3>
+
+<p>As of 1.10.12, users can launch pods via YAML files instead of
passing various configurations.</p>
+
+<p>To allow greater flexibility we have deprecated Airflow&rsquo;s
Pod class and instead now use classes and
+objects from the official Kubernetes API. The POD class will still work but
raise a deprecation
+warning. This feature involved a pretty extensive rewrite of all of our pod
creation code.</p>
+
+<p>Initially, we were going to hold off on these features until Airflow
2.0. However, we soon
+realized that exposing these features in 1.10.x is crucial in preparing users
for the 2.0 release to come.</p>
+
+<p>Details: <a
href="https://github.com/apache/airflow/pull/6230"
target="_blank">https://github.com/apache/airflow/pull/6230</a>
(<a
href="https://github.com/apache/airflow/commit/7aa0f472b57985a952a3e3d0a38f1b2535d93413"
target="_blank">Backport commit</a>)</p>
+
+<h2 id="updating-guide">Updating Guide</h2>
+
+<p>If you are updating Apache Airflow from a previous version to
<code>1.10.12</code>, please take a note of the following:</p>
+
+<ul>
+<li><p>Run <code>airflow upgradedb</code> after
<code>pip install -U apache-airflow==1.10.12</code> as
<code>1.10.12</code> contains 1 database
migration.</p></li>
+
+<li><p>As of airflow 1.10.12, using the
<code>airflow.contrib.kubernetes.Pod</code> class in the
<code>pod_mutation_hook</code> is now
+deprecated. Instead we recommend that users treat the pod parameter as a
<code>kubernetes.client.models.V1Pod</code> object.
+This means that users now have access to the full Kubernetes API when
modifying airflow pods for mutating POD.</p></li>
+
+<li><p>Previously, when tasks skipped by SkipMixin (such as
<code>BranchPythonOperator</code>,
<code>BaseBranchOperator</code> and
+<code>ShortCircuitOperator</code>) are cleared, they execute.
Since 1.10.12, when such skipped tasks are cleared,
+they will be skipped again by the newly introduced
<code>NotPreviouslySkippedDep</code>.</p></li>
+</ul>
+
+<h2 id="special-note">Special Note</h2>
+
+<h3 id="python-2">Python 2</h3>
+
+<p>Python 2 has reached end of its life on Jan 2020. Airflow Master no
longer supports Python 2.
+Airflow 1.10.* would be the last series to support Python 2.</p>
+
+<p>We strongly recommend users to use Python &gt;= 3.6</p>
+
+<h3 id="use-airflow-rbac-ui">Use Airflow RBAC UI</h3>
+
+<p>Airflow 1.10.10 ships with 2 UIs, the default is non-RBAC Flask-admin
based UI and Flask-appbuilder based UI.</p>
+
+<p>The Flask-AppBuilder (FAB) based UI allows Role-based Access Control
and has more advanced features compared to
+the legacy Flask-admin based UI. This UI can be enabled by setting
<code>rbac=True</code> in <code>[webserver]</code>
section in
+your <code>airflow.cfg</code>.</p>
+
+<p>Flask-admin based UI is deprecated and new features won&rsquo;t
be ported to it. This UI will still be the default
+for 1.10.* series but would no longer be available from Airflow 2.0</p>
+
+<h3 id="we-have-moved-to-github-issues">We have moved to Github
Issues</h3>
+
+<p>The Airflow Project has moved from <a
href="https://issues.apache.org/jira/projects/AIRFLOW/issues"
target="_blank">JIRA</a> to
+<a href="https://github.com/apache/airflow/issues"
target="_blank">Github</a> for tracking issues.</p>
+
+<p>So if you find any bugs in Airflow 1.10.10 please create a Github
Issue for it.</p>
+
+<h2 id="list-of-contributors">List of Contributors</h2>
+
+<p>According to git shortlog, the following people contributed to the
1.10.10 release. Thank you to all contributors!</p>
+
+<p>Alexander Sutcliffe, Andy, Aneesh Joseph, Ash Berlin-Taylor, Aviral
Agrawal, BaoshanGu, Beni Ben zikry,
+Daniel Imberman, Daniel Standish, Danylo Baibak, Ephraim Anierobi, Felix
Uellendall, Greg Neiheisel,
+Hartorn, Jacob Ferriero, Jannik F, Jarek Potiuk, Jinhui Zhang, Kamil Breguła,
Kaxil Naik, Kurganov,
+Luis Magana, Max Arrich, Pete DeJoy, Sumit Maheshwari, Tomek Urbaszek, Vicken
Simonian, Vinnie Guimaraes,
+William Tran, Xiaodong Deng, YI FU, Zikun Zhu, dewaldabrie, pulsar314,
retornam, yuqian90</p>
+
+ </description>
+ </item>
+
+ <item>
<title>Blog: Apache Airflow For Newcomers</title>
<link>/blog/apache-airflow-for-newcomers/</link>
<pubDate>Mon, 17 Aug 2020 00:00:00 +0000</pubDate>
diff --git a/search/index.html b/search/index.html
index 9b9e9c7..fa32f56 100644
--- a/search/index.html
+++ b/search/index.html
@@ -35,12 +35,12 @@
<meta property="og:url" content="/search/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Search Results">
<meta itemprop="description" content="">
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="0">
diff --git a/sitemap.xml b/sitemap.xml
index a90c9a3..940d201 100644
--- a/sitemap.xml
+++ b/sitemap.xml
@@ -4,207 +4,212 @@
<url>
<loc>/docs/overview/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/beds/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/ponycopters/configuring-ponycopters/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/getting-started/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/examples/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/ponycopters/launching-ponycopters/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tutorials/multi-bear/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/porridge/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/concepts/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/task/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tutorials/tutorial2/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tutorials/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/reference/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/contribution-guidelines/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
- <loc>/blog/apache-airflow-for-newcomers/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <loc>/blog/airflow-1.10.12/</loc>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
- <loc>/blog/tags/community/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <loc>/blog/tags/release/</loc>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/tags/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
+ </url>
+
+ <url>
+ <loc>/blog/apache-airflow-for-newcomers/</loc>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
+ </url>
+
+ <url>
+ <loc>/blog/tags/community/</loc>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/announcements/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/implementing-stable-api-for-apache-airflow/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/tags/rest-api/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/airflow-1.10.10/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
- </url>
-
- <url>
- <loc>/blog/tags/release/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/airflow-1.10.8-1.10.9/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/tags/documentation/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/experience-in-google-season-of-docs-2019-with-apache-airflow/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/airflow-survey/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/announcing-new-website/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/tags/survey/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/tags/users/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/apache-con-europe-2019-thoughts-and-insights-by-airflow-committers/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/tags/development/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/documenting-using-local-development-environments/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/its-a-breeze-to-develop-apache-airflow/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/getting-started/example-page/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/reference/parameter-reference/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/docs/tasks/ponycopters/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/use-cases/adobe/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/use-cases/big-fish-games/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/blog/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
@@ -213,62 +218,62 @@
<url>
<loc>/community/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/use-cases/dish/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/ecosystem/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/use-cases/experity/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/install/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/meetups/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/use-cases/onefootball/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/privacy-notice/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/roadmap/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/search/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
<url>
<loc>/use-cases/</loc>
- <lastmod>2020-08-25T21:38:49+01:00</lastmod>
+ <lastmod>2020-08-26T18:59:09+01:00</lastmod>
</url>
</urlset>
\ No newline at end of file
diff --git a/tags/index.html b/tags/index.html
index d53e221..ae68fcf 100644
--- a/tags/index.html
+++ b/tags/index.html
@@ -37,7 +37,7 @@
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="og:updated_time" content="2020-08-17T00:00:00+00:00" /><meta
property="og:site_name" content="Apache Airflow" />
+<meta property="og:updated_time" content="2020-08-25T00:00:00+00:00" /><meta
property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Tags">
<meta itemprop="description" content="Platform created by the community to
programmatically author, schedule and monitor workflows.">
diff --git a/tags/index.xml b/tags/index.xml
index 5eef421..b1ab28e 100644
--- a/tags/index.xml
+++ b/tags/index.xml
@@ -4,7 +4,7 @@
<link>/tags/</link>
<description>Recent content in Tags on Apache Airflow</description>
<generator>Hugo -- gohugo.io</generator>
- <lastBuildDate>Mon, 17 Aug 2020 00:00:00 +0000</lastBuildDate>
+ <lastBuildDate>Tue, 25 Aug 2020 00:00:00 +0000</lastBuildDate>
<atom:link href="/tags/index.xml" rel="self"
type="application/rss+xml" />
diff --git a/use-cases/adobe/index.html b/use-cases/adobe/index.html
index 40f6926..0bb1fd3 100644
--- a/use-cases/adobe/index.html
+++ b/use-cases/adobe/index.html
@@ -35,12 +35,12 @@
<meta property="og:url" content="/use-cases/adobe/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Adobe">
<meta itemprop="description" content="What was the problem? Modern big data
platforms need sophisticated data pipelines connecting to many backend services
enabling complex workflows. These workflows need to be deployed, monitored, and
run either on regular schedules or triggered by external events. Adobe
Experience Platform component services architected and built an orchestration
service to enable their users to author, schedule, and monitor complex
hierarchical (including sequential a [...]
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="251">
diff --git a/use-cases/big-fish-games/index.html
b/use-cases/big-fish-games/index.html
index dafc927..a598c05 100644
--- a/use-cases/big-fish-games/index.html
+++ b/use-cases/big-fish-games/index.html
@@ -35,12 +35,12 @@
<meta property="og:url" content="/use-cases/big-fish-games/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Big Fish Games">
<meta itemprop="description" content="What was the problem? The main challenge
is the lack of standardized ETL workflow orchestration tools. PowerShell and
Python-based ETL frameworks built in-house are currently used for scheduling
and running analytical workloads. However, there is no web UI through which we
can monitor these workflows and it requires additional effort to maintain this
framework. These scheduled jobs based on external dependencies are not well
suited to modern Big Data [...]
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="336">
diff --git a/use-cases/dish/index.html b/use-cases/dish/index.html
index 181bd4c..502b5a5 100644
--- a/use-cases/dish/index.html
+++ b/use-cases/dish/index.html
@@ -35,12 +35,12 @@
<meta property="og:url" content="/use-cases/dish/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Dish">
<meta itemprop="description" content="What was the problem? We faced
increasing complexity managing lengthy crontabs with scheduling being an issue,
this required carefully planning timing due to resource constraints, usage
patterns, and especially custom code needed for retry logic. In the last case,
having to verify success of previous jobs and/or steps prior to running the
next. Furthermore, time to results is important, but we were increasingly
relying on buffers for processing, wher [...]
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="249">
diff --git a/use-cases/experity/index.html b/use-cases/experity/index.html
index 6dc8412..0df2762 100644
--- a/use-cases/experity/index.html
+++ b/use-cases/experity/index.html
@@ -36,13 +36,13 @@ How did Apache Airflow help to solve this problem?
Ultimately we decided flexibl
<meta property="og:url" content="/use-cases/experity/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Experity">
<meta itemprop="description" content="What was the problem? We had to deploy
our complex, flagship app to multiple nodes in multiple ways. This required
tasks to communicate across Windows nodes and coordinate timing perfectly. We
did not want to buy an expensive enterprise scheduling tool and needed ultimate
flexibility.
How did Apache Airflow help to solve this problem? Ultimately we decided
flexible, multi-node, DAG capable tooling was key and airflow was one of the
few tools that fit that bill.">
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="191">
diff --git a/use-cases/onefootball/index.html b/use-cases/onefootball/index.html
index 1317e21..fff5e66 100644
--- a/use-cases/onefootball/index.html
+++ b/use-cases/onefootball/index.html
@@ -36,13 +36,13 @@ On top of that, new data tools appear each month: third
party data sources, clou
<meta property="og:url" content="/use-cases/onefootball/" />
<meta property="og:image" content="/images/feature-image.png" />
-<meta property="article:modified_time" content="2020-08-25T21:38:49+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
+<meta property="article:modified_time" content="2020-08-26T18:59:09+01:00"
/><meta property="og:site_name" content="Apache Airflow" />
<meta itemprop="name" content="Onefootball">
<meta itemprop="description" content="What was the problem? With millions of
daily active users, managing the complexity of data engineering at Onefootball
is a constant challenge. Lengthy crontabs, multiplication of custom API
clients, erosion of confidence in the analytics served, increasing heroism
(“only one person can solve this issue”). Those are the challenges
that most teams face unless they consciously invest in their tools and
processes.
On top of that, new data tools appear each month: third party data sources,
cloud providers solutions, different storage technologies… Managing all
those integrations is costly and brittle, especially for small data engineering
teams that are trying to do more with less.">
-<meta itemprop="dateModified" content="2020-08-25T21:38:49+01:00" />
+<meta itemprop="dateModified" content="2020-08-26T18:59:09+01:00" />
<meta itemprop="wordCount" content="294">
