This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 1647dedba57 Update documentation about publishing docs to staging
(#50568)
1647dedba57 is described below
commit 1647dedba571978ff09f1917e0199144f49f2256
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Wed May 14 16:45:08 2025 +0200
Update documentation about publishing docs to staging (#50568)
* Update documentation about publishing docs to staging
Fixes ##50567
* Update dev/README_RELEASE_AIRFLOW.md
Co-authored-by: Pierre Jeambrun <pierrejb...@gmail.com>
* fixup! Update dev/README_RELEASE_AIRFLOW.md
---------
Co-authored-by: Pierre Jeambrun <pierrejb...@gmail.com>
---
dev/README_RELEASE_AIRFLOW.md | 54 ++++++++++++++++-----
dev/README_RELEASE_HELM_CHART.md | 47 ++++++++++++++----
dev/README_RELEASE_PROVIDERS.md | 100 +++++++++++++++++++++++++++++++--------
docs/README.md | 43 ++++++++++++++---
docs/images/publish-site.png | Bin 24194 -> 0 bytes
docs/images/publish_site.png | Bin 0 -> 173681 bytes
6 files changed, 199 insertions(+), 45 deletions(-)
diff --git a/dev/README_RELEASE_AIRFLOW.md b/dev/README_RELEASE_AIRFLOW.md
index ebe8349a82b..6e426edab64 100644
--- a/dev/README_RELEASE_AIRFLOW.md
+++ b/dev/README_RELEASE_AIRFLOW.md
@@ -27,6 +27,7 @@
- [Prepare the Apache Airflow Package
RC](#prepare-the-apache-airflow-package-rc)
- [Update the milestone](#update-the-milestone)
- [Build RC artifacts](#build-rc-artifacts)
+ - [Publish release candidate documentation
(staging)](#publish-release-candidate-documentation-staging)
- [Prepare production Docker Image RC](#prepare-production-docker-image-rc)
- [Prepare Vote email on the Apache Airflow release
candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate)
- [Verify the release candidate by PMC
members](#verify-the-release-candidate-by-pmc-members)
@@ -42,7 +43,7 @@
- [Publish release to SVN](#publish-release-to-svn)
- [Manually prepare production Docker
Image](#manually-prepare-production-docker-image)
- [Verify production images](#verify-production-images)
- - [Publish documentation](#publish-documentation)
+ - [Publish final documentation](#publish-final-documentation)
- [Notify developers of release](#notify-developers-of-release)
- [Send announcements about security issues fixed in the
release](#send-announcements-about-security-issues-fixed-in-the-release)
- [Add release data to Apache Committee Report
Helper](#add-release-data-to-apache-committee-report-helper)
@@ -330,6 +331,37 @@ pipx install -e ./dev/breeze
--current-release ${VERSION}
```
+## Publish release candidate documentation (staging)
+
+Documentation is an essential part of the product and should be made available
to users.
+In our cases, documentation for the pre-release versions is published in
staging S3 bucket.
+The documentation source code and build tools are available in the
`apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details
about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows
including manual workflows for
+emergency cases.
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to the staging S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
You should
+specify the RC tag and choose `s3://staging-docs-airflow-apache-org/docs`
bucket as destination
+
+You should specify 'apache-airflow docker-stack' passed as packages to be
+built.
+
+After that step, the provider documentation should be available under
https://airflow.stage.apache.org//
+URL (RC PyPI packages are build with the staging urls) but stable links and
drop-down boxes are not updated yet.
+
+2. Invalidate Fastly cache, update version drop-down and stable links with the
new versions of the documentation.
+
+In order to do it, you need to run the [Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository - but make sure to use `staging` branch.
+
+After that workflow completes, the new version should be available in the
drop-down list and stable links
+should be updated, also Fastly cache will be updated
+
## Prepare production Docker Image RC
Production Docker images should be manually prepared and pushed by the release
manager or another committer
@@ -815,10 +847,10 @@ docker pull apache/airflow:${VERSION}
breeze prod-image verify --image-name apache/airflow:${VERSION}
```
-## Publish documentation
+## Publish final documentation
Documentation is an essential part of the product and should be made available
to users.
-In our cases, documentation for the released versions is published in S3
bucket, and the site is
+In our cases, documentation for the released versions is published in the live
S3 bucket, and the site is
kept in a separate repository -
[`apache/airflow-site`](https://github.com/apache/airflow-site),
but the documentation source code and build tools are available in the
`apache/airflow` repository, so
you need to run several workflows to publish the documentation. More details
about it can be found in
@@ -832,20 +864,20 @@ There are two steps to publish the documentation:
The release manager publishes the documentation using GitHub Actions workflow
[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
-You can specify the tag to use to build the docs and 'apache-airflow
docker-stack' passed as packages to be
-built.
+You should specify the final tag to build the docs and 'apache-airflow
docker-stack' should be
+passed as packages to be built. The Live bucket should be selected as
destination location.
-After that step, the provider documentation should be available under the
usual urls (same as in PyPI packages)
-but stable links and drop-down boxes should not be updated.
+After that step, the provider documentation should be available under the
https://airflow.apache.org/ URL
+(also linked directly from the PyPI packages) but stable links and drop-down
boxes should not be yet updated.
+That allows the Release Manager to verify if the documentation is published.
-2. Update version drop-down and stable links with the new versions of the
documentation.
+2. Invalidate Fastly Cache, update version drop-down and stable links with the
new versions of the documentation.
In order to do it, you need to run the [Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
-workflow in `airflow-site` repository.
+workflow in `airflow-site` repository. Make sure to use `main` as the branch
to run the workflow.
After that workflow completes, the new version should be available in the
drop-down list and stable links
-should be updated.
-
+should be updated, also Fastly cache will be invalidated.
## Notify developers of release
diff --git a/dev/README_RELEASE_HELM_CHART.md b/dev/README_RELEASE_HELM_CHART.md
index 2946da45761..efc55caf920 100644
--- a/dev/README_RELEASE_HELM_CHART.md
+++ b/dev/README_RELEASE_HELM_CHART.md
@@ -27,6 +27,7 @@
- [Build Release Notes](#build-release-notes)
- [Update minimum version of
Kubernetes](#update-minimum-version-of-kubernetes)
- [Build RC artifacts](#build-rc-artifacts)
+ - [Publish rc documentation](#publish-rc-documentation)
- [Prepare issue for testing status of
rc](#prepare-issue-for-testing-status-of-rc)
- [Prepare Vote email on the Apache Airflow release
candidate](#prepare-vote-email-on-the-apache-airflow-release-candidate)
- [Verify the release candidate by PMC
members](#verify-the-release-candidate-by-pmc-members)
@@ -40,7 +41,7 @@
- [Summarize the voting for the
release](#summarize-the-voting-for-the-release)
- [Publish release to SVN](#publish-release-to-svn)
- [Publish release tag](#publish-release-tag)
- - [Publish documentation](#publish-documentation)
+ - [Publish final documentation](#publish-final-documentation)
- [Notify developers of release](#notify-developers-of-release)
- [Send announcements about security issues fixed in the
release](#send-announcements-about-security-issues-fixed-in-the-release)
- [Add release data to Apache Committee Report
Helper](#add-release-data-to-apache-committee-report-helper)
@@ -280,6 +281,36 @@ popd
git push apache tag helm-chart/${VERSION}${VERSION_SUFFIX}
```
+## Publish rc documentation
+
+Documentation is an essential part of the product and should be made available
to users.
+In our cases, documentation for the released versions is published in S3
bucket, and the site is
+kept in a separate repository -
[`apache/airflow-site`](https://github.com/apache/airflow-site),
+but the documentation source code and build tools are available in the
`apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details
about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows
including manual workflows for
+emergency cases.
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to the staging S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+
+You can specify the RC tag to use to build the docs and 'helm-chart' passed as
packages to be built.
+
+After that step, the provider documentation should be available under the
https://airflow.staged.apache.org
+(same as in the helm chart documentation).
+
+2. Invalidate Fastly cache for the documentation.
+
+In order to do it, you need to run the [Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository. Make sure to use `staging` branch.
+
+After that workflow completes, the new version should be available in the
drop-down list and stable links
+should be updated and Fastly cache should be invalidated.
+
## Prepare issue for testing status of rc
Create an issue for testing status of the RC (PREVIOUS_RELEASE should be the
previous release version
@@ -684,7 +715,7 @@ git tag -s helm-chart/${VERSION} -m "Apache Airflow Helm
Chart ${VERSION}"
git push apache helm-chart/${VERSION}
```
-## Publish documentation
+## Publish final documentation
Documentation is an essential part of the product and should be made available
to users.
In our cases, documentation for the released versions is published in S3
bucket, and the site is
@@ -696,23 +727,23 @@ emergency cases.
There are two steps to publish the documentation:
-1. Publish the documentation to S3 bucket.
+1. Publish the documentation to the live S3 bucket.
The release manager publishes the documentation using GitHub Actions workflow
[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
You can specify the tag to use to build the docs and 'helm-chart' passed as
packages to be built.
-After that step, the provider documentation should be available under the
usual urls (same as in PyPI packages)
-but stable links and drop-down boxes should not be updated.
+After that step, the provider documentation should be available under the
https://airflow.apache.org
+(same as in the helm chart documentation).
-2. Update version drop-down and stable links with the new versions of the
documentation.
+2. Invalidated Fastly cache for the documentation.
In order to do it, you need to run the [Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
-workflow in `airflow-site` repository.
+workflow in `airflow-site` repository. Make sure to use `main` branch.
After that workflow completes, the new version should be available in the
drop-down list and stable links
-should be updated.
+should be updated and Fastly cache should be invalidated.
## Notify developers of release
diff --git a/dev/README_RELEASE_PROVIDERS.md b/dev/README_RELEASE_PROVIDERS.md
index 014750550fc..f3f507b9de5 100644
--- a/dev/README_RELEASE_PROVIDERS.md
+++ b/dev/README_RELEASE_PROVIDERS.md
@@ -35,8 +35,9 @@
- [Build Provider distributions for SVN apache
upload](#build-provider-distributions-for-svn-apache-upload)
- [Build and sign the source and convenience
packages](#build-and-sign-the-source-and-convenience-packages)
- [Commit the source packages to Apache SVN
repo](#commit-the-source-packages-to-apache-svn-repo)
- - [Publish the Regular convenience package to
PyPI](#publish-the-regular-convenience-package-to-pypi)
+ - [Publish the Regular distributions to PyPI (release
candidates)](#publish-the-regular-distributions-to-pypi-release-candidates)
- [Add tags in git](#add-tags-in-git)
+ - [Publish release candidate
documentation](#publish-release-candidate-documentation)
- [Prepare issue in GitHub to keep status of
testing](#prepare-issue-in-github-to-keep-status-of-testing)
- [Prepare voting email for Providers release
candidate](#prepare-voting-email-for-providers-release-candidate)
- [Verify the release candidate by PMC
members](#verify-the-release-candidate-by-pmc-members)
@@ -46,7 +47,7 @@
- [Publish release to SVN](#publish-release-to-svn)
- [Publish the packages to PyPI](#publish-the-packages-to-pypi)
- [Add tags in git](#add-tags-in-git-1)
- - [Publish documentation](#publish-documentation)
+ - [Publish final documentation](#publish-final-documentation)
- [Update providers metadata](#update-providers-metadata)
- [Notify developers of release](#notify-developers-of-release)
- [Send announcements about security issues fixed in the
release](#send-announcements-about-security-issues-fixed-in-the-release)
@@ -414,7 +415,7 @@ svn rm file_name // repeate that for every file
svn commit -m "delete old providers"
```
-## Publish the Regular convenience package to PyPI
+## Publish the Regular distributions to PyPI (release candidates)
In order to publish release candidate to PyPI you just need to build and
release packages.
The packages should however contain the rcN suffix in the version file name
but not internally in the package,
@@ -485,6 +486,63 @@ If you want to disable this behaviour, set the env
**CLEAN_LOCAL_TAGS** to false
breeze release-management tag-providers
```
+## Publish release candidate documentation
+
+Documentation is an essential part of the product and should be made available
to users.
+In our cases, documentation for the released versions is published in the
staging S3 bucket, and the site is
+kept in a separate repository -
[`apache/airflow-site`](https://github.com/apache/airflow-site),
+but the documentation source code and build tools are available in the
`apache/airflow` repository, so
+you need to run several workflows to publish the documentation. More details
about it can be found in
+[Docs README](../docs/README.md) showing the architecture and workflows
including manual workflows for
+emergency cases.
+
+There are two steps to publish the documentation:
+
+1. Publish the documentation to the staging S3 bucket.
+
+The release manager publishes the documentation using GitHub Actions workflow
+[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
+
+You should specify the final tag to use to build the docs and list of
providers to publish
+(separated by spaces) or ``all-providers`` in case you want to publish all
providers
+(optionally you can exclude some of those providers). You should use `staging`
bucket to publish the release
+candidate documentation.
+
+After that step, the provider documentation should be available under the
http://airflow.staged.apache.org URL
+(also present in the PyPI packages) but stable links and drop-down boxes
should not be yet updated.
+
+2. Invalidate Fastly cache, update version drop-down and stable links with the
new versions of the documentation.
+
+Before doing it - review the state of removed, suspended, new packages in
+[the docs
index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md):
+Make sure to use `staging` branch to run the workflow.
+
+There are few special considerations when the list of provider is updated.
+
+- If you publish a new package, you must add it to the list of packages in the
index.
+- If there are changes to suspension or removal status of a package, you must
move it appropriate section.
+
+- In case you need to make any changes - create the commit and push changes
and merge it to `staging` branch.
+ in [airflow-site](https://github.com/apache/airflow-site) repository.
+
+```shell script
+cd "${AIRFLOW_SITE_DIRECTORY}"
+branch="add-documentation-$(date "+%Y-%m-%d%n")"
+git checkout -b "${branch}"
+git add .
+git commit -m "Add documentation for packages - $(date "+%Y-%m-%d%n")"
+git push --set-upstream origin "${branch}"
+```
+
+Merging the PR with the index changes to `staging` will trigger site
publishing.
+
+If you do not need to merge a PR, you should manually run the
+[Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
+workflow in `airflow-site` repository to refresh indexes and drop-downs.
+
+After that build from PR or workflow completes, the new version should be
available in the drop-down
+list and stable links should be updated, also Fastly cache will be invalidated.
+
## Prepare issue in GitHub to keep status of testing
Create a GitHub issue with the content generated via manual execution of the
command below. You will use
@@ -1136,10 +1194,10 @@ If you want to disable this behaviour, set the env
**CLEAN_LOCAL_TAGS** to false
breeze release-management tag-providers
```
-## Publish documentation
+## Publish final documentation
Documentation is an essential part of the product and should be made available
to users.
-In our cases, documentation for the released versions is published in S3
bucket, and the site is
+In our cases, documentation for the released versions is published in the live
S3 bucket, and the site is
kept in a separate repository -
[`apache/airflow-site`](https://github.com/apache/airflow-site),
but the documentation source code and build tools are available in the
`apache/airflow` repository, so
you need to run several workflows to publish the documentation. More details
about it can be found in
@@ -1148,29 +1206,31 @@ emergency cases.
There are two steps to publish the documentation:
-1. Publish the documentation to S3 bucket.
+1. Publish the documentation to the live S3 bucket.
The release manager publishes the documentation using GitHub Actions workflow
[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
-You can specify the tag to use to build the docs and list of providers to
publish (separated by spaces) or
-``all-providers`` in case you want to publish all providers (optionally you
can exclude
-some of those providers)
+You should specify the final tag to use to build the docs and list of
providers to publish
+(separated by spaces) or ``all-providers`` in case you want to publish all
providers
+(optionally you can exclude some of those providers). You should use `live`
bucket to publish the final
+release documentation.
-After that step, the provider documentation should be available under the
usual urls (same as in PyPI packages)
-but stable links and drop-down boxes should not be updated.
+After that step, the provider documentation should be available under the
http://airflow.apache.org URL
+(also present in the PyPI packages) but stable links and drop-down boxes
should not be yet updated.
-Use "live" S3 to publish the final documentation.
-
-2. Update version drop-down and stable links with the new versions of the
documentation.
+2. Invalidate Fastly cache, update version drop-down and stable links with the
new versions of the documentation.
Before doing it - review the state of removed, suspended, new packages in
[the docs
index](https://github.com/apache/airflow-site/blob/master/landing-pages/site/content/en/docs/_index.md):
+Make sure to use `main` branch to run the workflow.
+
+There are few special considerations when the list of provider is updated.
- If you publish a new package, you must add it to the list of packages in the
index.
-- If there are changes to suspension or removal status of a package you must
move it appropriate section.
+- If there are changes to suspension or removal status of a package, you must
move it appropriate section.
-- In case you need to make any changes - create the commit and push changes
and merge it to main branch.
+- In case you need to make any changes - create the commit and push changes
and merge it to `main` branch.
in [airflow-site](https://github.com/apache/airflow-site) repository.
```shell script
@@ -1182,14 +1242,14 @@ git commit -m "Add documentation for packages - $(date
"+%Y-%m-%d%n")"
git push --set-upstream origin "${branch}"
```
-Merging the PR with the index changes to main will trigger site publishing.
+Merging the PR with the index changes to `main` will trigger site publishing.
-If you do not need to merge a PR, you can manually run the
+If you do not need to merge a PR, you should manually run the
[Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
-workflow in `airflow-site` repository.
+workflow in `airflow-site` repository to refresh indexes and drop-downs.
After that build from PR or workflow completes, the new version should be
available in the drop-down
-list and stable links should be updated.
+list and stable links should be updated, also Fastly cache will be invalidated.
## Update providers metadata
diff --git a/docs/README.md b/docs/README.md
index 7654e64270a..59f0bd379ef 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -24,6 +24,7 @@
- [Documentation configuration](#documentation-configuration)
- [Architecture of documentation for
Airflow](#architecture-of-documentation-for-airflow)
- [Diagrams of the documentation
architecture](#diagrams-of-the-documentation-architecture)
+- [Staging site](#staging-site)
- [Typical workflows](#typical-workflows)
- [Publishing the documentation by the release
manager](#publishing-the-documentation-by-the-release-manager)
- [Publishing changes to the website (including
theme)](#publishing-changes-to-the-website-including-theme)
@@ -85,7 +86,22 @@ This is the diagram of live documentation architecture:
-Staging documentation architecture is similar, but uses staging bucket and
staging Apache Website.
+# Staging site
+
+Staging documentation architecture is similar, but uses staging bucket and
staging Apache Website. The main
+differences are:
+
+* The staging bucket is `s3://staging-docs-airflow-apache-org/docs/`
+* The staging website is `https://airflow.staged.apache.org/docs/`
+* The staging site is deployed by merging PR or pushing `staging` branch in
the `airflow-site` repository
+ rather than main. The `staging` branch should be periodically rebased to the
`main` branch, but while
+ some changes are developed in `staging` it can diverge from `main` branch.
+* Merging into `staging` branch of `airflow-site` repository or pushing
`staging` branch
+ will automatically trigger the build of the website and publish it to the
`publish-staging` branch and
+ effectively to the staging site.
+
+Documentation of pre-release versions of Airflow distributions should be
published to the staging s3
+bucket so that we can test the documentation before we publish it to the live
bucket.
# Typical workflows
@@ -97,6 +113,13 @@ The release manager publishes the documentation using
GitHub Actions workflow
[Publish Docs to
S3](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml).
The same workflow can be used to publish Airflow, Helm chart and providers
documentation.
+This workflow is used twice:
+
+* when pre-release distributions are prepared (alpha/beta/rc) - the
documentation should be published to
+ the staging bucket and staging site should be built and published.
+* when final releases of distributions are prepared - the documentation should
be published to the live
+ bucket and the live website should be built and published.
+
The person who triggers the build (release manager) should specify the tag
name of the docs to be published
and the list of documentation packages to be published. Usually it is:
@@ -108,7 +131,9 @@ Optionally - specifically if we run `all-providers` and
release manager wants to
they can specify documentation packages to exclude. Leaving "no-docs-excluded"
will publish all packages
specified to be published without exclusions.
-You can also specify whether "live" or "staging" documentation should be
published. The default is "live".
+You can also specify whether "live" or "staging" documentation should be
published. The default is "live". In
+the future we might add `auto` setting that will publish to "live" or
"staging" depending on the tag
+used to trigger the workflow.
Example screenshot of the workflow triggered from the GitHub UI:
@@ -120,14 +145,20 @@ with direct URL (say
https://apache.airflow.org/docs/apache-airflow/3.0.1/) but
point to previous version of the documentation as `stable` and the version
drop-downs will not be updated.
In order to do it, you need to run the [Build
docs](https://github.com/apache/airflow-site/actions/workflows/build.yml)
-workflow in `airflow-site` repository. This will build the website and publish
it to the `publish`
-branch of `airflow-site` repository, including refreshing of the version
numbers in the drop-downs and
-stable links.
+workflow in `airflow-site` repository.
+
+For `live` site you should run the workflow in `main` branch. For `staging`
site it should be staging branch.
+This will build the website and publish it to the `publish` branch of
`airflow-site` repository (for `live`
+site) or `publish-staging` branch, (for `staging` site). The workflow will
also update the website with
+including refreshing of the version numbers in the drop-downs and stable links.
+This workflow also invalidates cache in Fastly that Apache Software Foundation
uses to serve the website,
+so you should always run it after you modify the documentation for the
website. Other than that Fastly is
+configured with 3600 seconds TTL - which means that changes will propagate to
the website in ~1 hour.
-Some time after the workflow succeeds and documentation is published, in live
bucket, the `airflow-site-archive`
+Shortly after the workflow succeeds and documentation is published, in live
bucket, the `airflow-site-archive`
repository is automatically synchronized with the live S3 bucket. TODO:
IMPLEMENT THIS, FOR NOW IT HAS
TO BE MANUALLY SYNCHRONIZED VIA [Sync s3 to
GitHub](https://github.com/apache/airflow-site-archive/actions/workflows/s3-to-github.yml)
workflow in `airflow-site-archive` repository. The `airflow-site-archive`
essentially keeps the history of
diff --git a/docs/images/publish-site.png b/docs/images/publish-site.png
deleted file mode 100644
index f82d99df365..00000000000
Binary files a/docs/images/publish-site.png and /dev/null differ
diff --git a/docs/images/publish_site.png b/docs/images/publish_site.png
new file mode 100644
index 00000000000..0a6e7b89a8d
Binary files /dev/null and b/docs/images/publish_site.png differ
