Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2058202359 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Sure Noted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2058204641 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: We are not THERE yet - but we definitely plan to make step-by-step docs on how to publish - we also have few more improvements to our doc publishing process that should make it even easier and faster. This is part of https://github.com/apache/airflow/issues/44373 and this is but a part of https://github.com/apache/airflow/issues/45641 :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
kaxil commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2058004068 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Cool, thanks both -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
kaxil commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2058004934 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Worth adding some docs on https://github.com/apache/airflow-site when we get to the .htaccess part -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk merged PR #47814: URL: https://github.com/apache/airflow/pull/47814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2825464767 Some conflicts to resolve. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2056817728 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Yes. The idea here is that: * we build provider (and airflow docs) locally * we push them to s3 (only latest versions - all the rest is already in s3) * the change we push to "airflow-site" are just rewrite rules for the new versions in s3 + code to populate versions in the dropdwn -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2056788969 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: >Or is this for the docs on main? This docs publishing for released artifacts. it requires input of tag or branch then it generates docs and publishes to s3. >or is this for archival of the old docs? Archival we can add separate process inside aws, once we start serving docs from s3 and we can archive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2056790231 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Hope that make sense or am i making too confusing, @potiuk can pitch in :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2056788969 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: >Or is this for the docs on main? This docs publishing for released artifacts. >or is this for archival of the old docs? Archival we can add separate process inside aws, once we start serving docs from s3 and we can archive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r205606 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Yeah it will be served directly from s3 all the docs. we plan to use the .htaccess file rewrite rules to redirect. so provider and airflow docs are served from the s3 and rest are from the gh-pages branches. An example where i have tested with https://github.com/apache/airflow-site/blob/airflow-test/.htaccess and staging website https://airflow-test.staged.apache.org/ which serves docs from s3. Some discussions with @ashb and @potiuk are here https://apache-airflow.slack.com/archives/CJ1LVREHX/p1737287674851989?thread_ts=1737113729.883969&cid=CJ1LVREHX We could also add another steps later for archive where we can remove versions from serving bucket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r205606 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: Yeah it will be served directly from s3 all the docs. we plan to use the .htaccess(thanks to ash and jarek) file rewrite rules to redirect. so provider and airflow docs are served from the s3 and rest are from the gh-pages branches. An example where i have tested with https://github.com/apache/airflow-site/blob/airflow-test/.htaccess and staging website https://airflow-test.staged.apache.org/ which serves docs from s3. Some discussions with @ashb and @potiuk are here https://apache-airflow.slack.com/archives/CJ1LVREHX/p1737287674851989?thread_ts=1737113729.883969&cid=CJ1LVREHX We could also add another steps later for archive where we can remove versions from serving bucket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
kaxil commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r2056759785 ## dev/breeze/doc/09_release_management_tasks.rst: ## @@ -713,3 +713,57 @@ default is to build ``both`` type of distributions ``sdist`` and ``wheel``. :target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/doc/images/output_release-management_prepare-airflow-ctl-distributions.svg :width: 100% :alt: Breeze release-management prepare-airflow-ctl-distributions + +Publishing the documentation to S3 +"" + +To publish the documentation generated by ``build-docs`` in Breeze to ``S3``, +use the ``release-management publish-docs-to-s3`` command: + +.. code-block:: bash + + breeze release-management publish-docs-to-s3 + +The documentation publish to S3 should be done after the ``breeze release-management publish-docs`` command. +Once documentation is available in ``docs-archive`` directory of airflow-site, it can be published to S3. + +The publishing documentation to S3 consists of the following steps: + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path <> --destination-location <> + +Where ``--source-dir-path`` is a doc-archive location path and ``--destination-location`` is the S3 bucket path. + +.. code-block:: bash + + breeze release-management publish-docs --source-dir-path /User/pavan/airflow-site/docs-archive + --destination-location s3://airflow-docs/docs + +To exclude any documentation from publishing to S3, you can use the ``--exclude`` flag. + +.. code-block:: bash Review Comment: To clarify, the docs will be stored in S3, but served to the users how? Or is this for the docs on main? or is this for archival of the old docs? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2825297734 This looks really good. We should likely merge it and prepare a draft documentation for the next steps of doing it by @eladkal with the next provider's wave. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2824901149 Just updated few updates related to AIRFLOW_ROOT_PATH and some ordering of steps. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2822098623 > Happy to review / get back on this after 3.0.0 -- just too fried right now, sry! Yeah fine :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
kaxil commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2816638308 Happy to review / get back on this after 3.0.0 -- just too fried right now, sry! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
bugraoz93 commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2816592138 > > > Can i have some more review on this please? > > > > Looks great! Small question, are we planning to integrate this workflow in `ci.yml`? > > Not configured for CI, these new workflow will be triggered manually by release managers to publish docs to s3 when any release happens. :) > Thanks a lot for the details! Makes sense :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2816274785 > > Can i have some more review on this please? > > Looks great! Small question, are we planning to integrate this workflow in `ci.yml`? Not configured for CI, these new workflow will be triggered manually by release managers to publish docs to s3 when any release happens. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
bugraoz93 commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2816231554 > Can i have some more review on this please? Looks great! Small question, are we planning to integrate this workflow in `ci.yml`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2816020213 Can i have some more review on this please? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2810707714 > I am not super familiar with the process of doc publishing so dont take my word for it but LGTM :) Thanks vincent, no problem, this can wait, i will wait for others too review. :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2787935170 > Will rebase this and update after this merge #48760 Feel free! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2781471573 Will rebase this and update after this merge #48760 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2742492885 > I will take a look shortly - after finishing airflow-core one :) Sure, probably I will wait and refactor after big fish ( airflow-core) :) merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
vincbeck commented on code in PR #47814:
URL: https://github.com/apache/airflow/pull/47814#discussion_r2018808826
##
dev/breeze/src/airflow_breeze/utils/publish_docs_to_s3.py:
##
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from functools import cached_property
+
+import boto3
+
+from airflow_breeze.utils.console import get_console
+from airflow_breeze.utils.parallel import check_async_run_results,
run_with_pool
+
+PROVIDER_NAME_FORMAT = "apache-airflow-providers-{}"
+
+NON_SHORT_NAME_PACKAGES = ["docker-stack", "helm-chart", "apache-airflow"]
+
+s3_client = boto3.client("s3")
+
+
+class S3DocsPublish:
+def __init__(
+self,
+source_dir_path: str,
+destination_location: str,
+exclude_docs: str,
+dry_run: bool = False,
+overwrite: bool = False,
+parallelism: int = 1,
+):
+self.source_dir_path = source_dir_path
+self.destination_location = destination_location
+self.exclude_docs = exclude_docs
+self.dry_run = dry_run
+self.overwrite = overwrite
+self.parallelism = parallelism
+self.source_dest_mapping: list[tuple[str, str]] = []
+
+@cached_property
+def get_all_docs(self):
+get_console().print(f"[info]Getting all docs from
{self.source_dir_path}\n")
+try:
+all_docs = os.listdir(self.source_dir_path)
+except FileNotFoundError:
+get_console().print(f"[error]No docs found in
{self.source_dir_path}\n")
+sys.exit(1)
+return all_docs
+
+@cached_property
+def get_all_excluded_docs(self):
+if not self.exclude_docs:
+return []
+return self.exclude_docs.split(",")
+
+@cached_property
+def get_all_eligible_docs(self):
+"""
+It excludes the docs that are in the exclude list
+"""
+non_eligible_docs = []
+
+for excluded_doc in self.get_all_excluded_docs:
+if excluded_doc in NON_SHORT_NAME_PACKAGES:
+non_eligible_docs.append(excluded_doc)
+continue
+
+for doc in self.get_all_docs:
+excluded_provider_name =
PROVIDER_NAME_FORMAT.format(excluded_doc.replace(".", "-"))
+if doc == excluded_provider_name:
+non_eligible_docs.append(doc)
+continue
+
+docs_to_process = list(set(self.get_all_docs) - set(non_eligible_docs))
+if not docs_to_process:
+get_console().print("[error]No eligible docs found, all docs are
excluded\n")
+sys.exit(1)
+
+return docs_to_process
+
+def is_doc_exists(self, s3_bucket_doc_location: str) -> bool:
Review Comment:
```suggestion
def doc_exists(self, s3_bucket_doc_location: str) -> bool:
```
##
dev/breeze/src/airflow_breeze/utils/publish_docs_to_s3.py:
##
@@ -0,0 +1,198 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from functools import cached_property
+
+import boto3
+
+from airflow_breeze.utils.console import get_console
+from airflow_breeze.utils.parallel import check_async_run_results,
run_with_pool
+
+PROVIDER_NAME_FORMAT = "apache-airflow-providers-{}"
+
+NON_SHORT_NAME_PACKAGES = ["docker-stack", "helm-chart", "apache-airflow"]
+
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2745379646 Big fish done. Rebase and I will take a closer look :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add S3 docs publishing scripts [airflow]
potiuk commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2735734363 I will take a look shortly - after finishing airflow-core one :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add s3 docs publishing scripts [airflow]
gopidesupavan commented on PR #47814: URL: https://github.com/apache/airflow/pull/47814#issuecomment-2726783362 Workflow testing require, will be able to do it only after merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add s3 docs publishing scripts [airflow]
gopidesupavan commented on code in PR #47814: URL: https://github.com/apache/airflow/pull/47814#discussion_r1997061672 ## .github/workflows/publish-docs-to-s3.yml: ## @@ -0,0 +1,204 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: Publish Docs to S3 +on: # yamllint disable-line rule:truthy + workflow_dispatch: +inputs: + ref: +description: "The branch or tag to checkout for the docs publishing" +required: true +type: string + exclude-docs: +description: "Comma separated list of docs to exclude" +required: false +default: "" +type: string + destination-location: +description: "The destination location in S3" +required: false +default: "s3://staging-docs-airflow-apache-org/docs" Review Comment: It requires for testing after that we can configure default live bucket. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
