kaxil commented on code in PR #50464: URL: https://github.com/apache/airflow/pull/50464#discussion_r2084151834
########## docs/README.md: ########## @@ -37,3 +54,189 @@ Documentation for general overview and summaries not connected with any specific * `docker-stack-docs` - documentation for Docker Stack' * `providers-summary-docs` - documentation for provider summary page + +# Architecture of documentation for Airflow + +Building documentation for Airflow is optimized for speed and for convenience workflows of the release +managers and committers who publish and fix the documentation - that's why it's a little complex, as we have +multiple repositories and multiple sources of the documentation involved. + +There are few repositories under `apache` organization which are used to build the documentation for Airflow: + +* `apache-airflow` - the repository with the code and the documentation sources for Airflow distributions, + provider distributions, providers summary and docker summary: [apache-airflow](https://github.com/apache/airflow) + from here we publish the documentation to S3 bucket where the documentation is hosted. +* `airflow-site` - the repository with the website theme and content where we keep sources of the website + structure, navigation, theme for the website [airflow-site](https://github.com/apache/airflow). From here + we publish the website to the ASF servers so they are publish as the [official website](https://airflow.apache.org) +* `airflow-site-archive` - here we keep the archived historical versions of the generated documentation + of all the documentation packages that we keep on S3. This repository is automatically synchronized from + the S3 buckets and is only used in case we need to perform a bulk update of historical documentation. Here only + generated `html`, `css`, `js` and `images` files are kept, no sources of the documentation are kept here. + +We have two S3 buckets where we can publish the documentation generated from `apache-airflow` repository: + +* `s3://live-docs-airflow-apache-org/docs/` - live, [official documentation](https://airflow.apache.org/docs/) +* `s3://staging-docs-airflow-apache-org/docs/` - staging documentation [official documentation](https://staging-airflow.apache.org/docs/) TODO: make it works + +# Diagrams of the documentation architecture + +This is the diagram od live documentation architecture: + + + +Staging documentation architecture is similar, but uses staging bucket and staging Apache Website. + +# Typical workflows + +There are a few typical workflows that we support: + +## Publishing the documentation by the release manager + +The release manager publishes the documentation using `Publish Docs to S3` GitHub Action (accessible +via [GitHub UI](https://github.com/apache/airflow/actions/workflows/publish-docs-to-s3.yml). The same workflow can be used to publish Airflow, Helm chart and providers documentation. + +The person who triggers the build (release manager) should specify the tag name of the docs to be published +and the list of documentation packages to be published. Usually it is: + +* Airflow: `apache-airflow docker-stack` (later we will add `airflow-ctl` and `task-sdk`) +* Helm chart: `helm-chart` +* Providers: `provider_id1 provider_id2` or `all providers` if all providers should be published. + +Optionally - specifically if we run `all-providers` and release manager wants to exclude some providers, +they can specify documentation packages to exclude. Leaving "no-docs-excluded" will publish all packages +specified to be published without exclusions. + +You can also specify whether "live" or "staging" documentation should be published. The default is "live". + +Example screenshot of the workflow triggered from the GitHub UI: + + + +Right after the workflow succeeds and documentation is published, in live bucket, the `airflow-site-archive` +repository is automatically synchronized with the live S3 bucket. TO DO: IMPLEMENT THIS, FOR NOW IT HAS Review Comment: ```suggestion repository is automatically synchronized with the live S3 bucket. TODO: IMPLEMENT THIS, FOR NOW IT HAS ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
