This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 4deed64132 Update release process for providers to include mixed RC
versions (#36385)
4deed64132 is described below
commit 4deed641322343cb18eabd9ad199f5ac83f29ff0
Author: Jarek Potiuk <[email protected]>
AuthorDate: Sat Dec 23 17:46:29 2023 +0100
Update release process for providers to include mixed RC versions (#36385)
This PR updates released process for providers to enable releasing
providers in more regular batches. Sometimes when we exclude a
provider from previous voting, we want to release RCN (2,3 etc.)
candidate.
However, especially when time between previous RC and the new one
is long (for example because fixing took a long time) we might
want to release the RCN release for that cancelled providers and
RC1 for all the providers that have been changed in the meantime.
This cchange makes it possible (and easy):
1) release RC1 for all providers (the RCN provider should be skipped,
because tag for this provider already exists.
2) release the RCN providers with `--version-suffix-for-pypi rcN`.
The release process and tools were updated to account for that - where
rc candidate number is retrieved from packages prepared in `dist`.
Fixed a few small missing things in the process.
---
dev/README_RELEASE_PROVIDER_PACKAGES.md | 20 ++++++++++++--------
.../commands/release_management_commands.py | 22 +++++++++++++++++++++-
.../provider_issue_TEMPLATE.md.jinja2 | 2 +-
3 files changed, 34 insertions(+), 10 deletions(-)
diff --git a/dev/README_RELEASE_PROVIDER_PACKAGES.md
b/dev/README_RELEASE_PROVIDER_PACKAGES.md
index 21e909e8e9..3f8fd3c448 100644
--- a/dev/README_RELEASE_PROVIDER_PACKAGES.md
+++ b/dev/README_RELEASE_PROVIDER_PACKAGES.md
@@ -330,13 +330,21 @@ breeze release-management prepare-provider-packages
--include-removed-providers
--version-suffix-for-pypi rc1 --package-format both
```
-if you only build few packages, run:
+If you only build few packages, run:
```shell script
breeze release-management prepare-provider-packages \
--version-suffix-for-pypi rc1 --package-format both PACKAGE PACKAGE ....
```
+In case you are ALSO releasing RC2, RC3, etc. for selected packages, they will
be skipped automatically because
+the `rc1` tag will be created for them already. In this case you should
specify the ``rc*`` that you want to
+build and specify the package id's you want to build.
+
+```shell script
+breeze release-management prepare-provider-packages --version-suffix-for-pypi
rc2 --package-format both PACKAGE
+```
+
* Verify the artifacts that would be uploaded:
```shell script
@@ -416,7 +424,7 @@ git pull --rebase
```shell script
cd "${AIRFLOW_REPO_ROOT}"
-breeze build-docs --clean-build apache-airflow-providers all-providers
+breeze build-docs --clean-build apache-airflow-providers all-providers
--include-removed-providers
```
Usually when we release packages we also build documentation for the
"documentation-only" packages. This
@@ -461,12 +469,6 @@ breeze release-management publish-docs
apache-airflow-providers all-providers --
breeze release-management add-back-references all-providers
```
-If you see `ModuleNotFoundError: No module named 'docs'`, set:
-
-```
-export PYTHONPATH=.:${PYTHONPATH}
-```
-
If you have providers as list of provider ids because you just released them
you can build them with
```shell script
@@ -503,6 +505,8 @@ You can also pass the token as `--github-token` option in
the script.
You can also pass list of PR to be excluded from the issue with
`--excluded-pr-list`.
```shell script
+cd "${AIRFLOW_REPO_ROOT}"
+
breeze release-management generate-issue-content-providers
--only-available-in-dist
```
diff --git
a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
index fe33c73ff2..a18a3ecd1c 100644
--- a/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/release_management_commands.py
@@ -1496,6 +1496,23 @@ def is_package_in_dist(dist_files: list[str], package:
str) -> bool:
)
+VERSION_MATCH = re.compile(r"([0-9]+)\.([0-9]+)\.([0-9]+)(.*)")
+
+
+def get_suffix_from_package_in_dist(dist_files: list[str], package: str) ->
str | None:
+ """Get suffix from package prepared in dist folder."""
+ for file in dist_files:
+ if file.startswith(f'apache_airflow_providers_{package.replace(".",
"_")}') and file.endswith(
+ ".tar.gz"
+ ):
+ file = file[: -len(".tar.gz")]
+ version = file.split("-")[-1]
+ match = VERSION_MATCH.match(version)
+ if match:
+ return match.group(4)
+ return None
+
+
def get_prs_for_package(provider_id: str) -> list[int]:
pr_matcher = re.compile(r".*\(#([0-9]*)\)``$")
prs = []
@@ -1567,6 +1584,7 @@ def generate_issue_content_providers(
pypi_package_name: str
version: str
pr_list: list[PullRequest.PullRequest | Issue.Issue]
+ suffix: str
if not provider_packages:
provider_packages = list(DEPENDENCIES.keys())
@@ -1623,16 +1641,18 @@ def generate_issue_content_providers(
).read_text()
)
if pull_request_list:
+ package_suffix =
get_suffix_from_package_in_dist(files_in_dist, provider_id)
providers[provider_id] = ProviderPRInfo(
version=provider_yaml_dict["versions"][0],
provider_package_id=provider_id,
pypi_package_name=provider_yaml_dict["package-name"],
pr_list=pull_request_list,
+ suffix=package_suffix if package_suffix else suffix,
)
template = jinja2.Template(
(Path(__file__).parents[1] /
"provider_issue_TEMPLATE.md.jinja2").read_text()
)
- issue_content = template.render(providers=providers,
date=datetime.now(), suffix=suffix)
+ issue_content = template.render(providers=providers,
date=datetime.now())
get_console().print()
get_console().print(
"[green]Below you can find the issue content that you can use "
diff --git a/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
b/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
index 1639a8857d..8009549772 100644
--- a/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
+++ b/dev/breeze/src/airflow_breeze/provider_issue_TEMPLATE.md.jinja2
@@ -10,7 +10,7 @@ Let us know in the comment, whether the issue is addressed.
Those are providers that require testing as there were some substantial
changes introduced:
{% for provider_id, provider_info in providers.items() %}
-## Provider [{{ provider_id }}: {{ provider_info.version }}{{ suffix
}}](https://pypi.org/project/{{ provider_info.pypi_package_name }}/{{
provider_info.version }}{{ suffix }})
+## Provider [{{ provider_id }}: {{ provider_info.version }}{{
provider_info.suffix }}](https://pypi.org/project/{{
provider_info.pypi_package_name }}/{{ provider_info.version }}{{
provider_info.suffix }})
{%- for pr in provider_info.pr_list %}
- [ ] [{{ pr.title }} (#{{ pr.number }})]({{ pr.html_url }}): @{{
pr.user.login }}
{%- endfor %}