Dev-iL commented on code in PR #1529:
URL: https://github.com/apache/hamilton/pull/1529#discussion_r3005725180
##########
scripts/apache_release_helper.py:
##########
@@ -574,29 +601,39 @@ def main():
if not files_to_upload:
sys.exit(1)
- # Upload artifacts
- print(f"\n{'=' * 80}")
- print(" Uploading to Apache SVN")
- print(f"{'=' * 80}\n")
- # NOTE: You MUST have your SVN client configured to use your Apache ID and
have permissions.
- svn_url = svn_upload(package_name, version, rc_num, files_to_upload,
apache_id)
- if not svn_url:
- sys.exit(1)
-
- # Generate email
- print(f"\n{'=' * 80}")
- print(" Vote Email Template")
- print(f"{'=' * 80}\n")
- generate_email_template(package_name, version, rc_num, svn_url)
+ if args.dry_run:
+ # Dry run: skip SVN upload, show summary
+ print(f"\n{'=' * 80}")
+ print(" [dry-run] Skipping SVN upload")
+ print(f"{'=' * 80}\n")
+ print("Artifacts built successfully:")
+ for f in files_to_upload:
+ print(f" {f}")
+ print("\nTo do a real release, re-run without --dry-run.")
+ else:
+ # Upload artifacts
Review Comment:
Have you considered
https://packaging.python.org/en/latest/guides/using-testpypi/?
##########
scripts/apache_release_helper.py:
##########
@@ -339,29 +357,28 @@ def create_release_artifacts(package_config: dict,
version) -> list[str]:
# Use -src suffix to distinguish source distribution from wheel
(convenience package)
new_tar_ball =
f"dist/{package_name}-{version.lower()}-incubating-src.tar.gz"
_modify_tarball_for_apache_release(tarball_path[0], new_tar_ball,
package_name)
+ # Remove original flit tarball (only keep the incubating copy)
+ os.remove(tarball_path[0])
Review Comment:
What's in position [0]? Is there a risk it's ever something else? Maybe use
`list.index`? Same applies to `wheel_file.`
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
Review Comment:
```suggestion
both the spirit and (when applicable) the letter.
```
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
+
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is:
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
+
+Specifically, we set the following guidelines:
+
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | Y | Y | N | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | Y | Y | Y | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (unit + plugin) | Y | Y | N | We expect users/PMC to download the
source distribution, build from source, run the tests, and validate. Thus we
include in the tar.gz and sdist, but not in the whl. |
+| READMEs | Y | Y | Y | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions. |
+| Documentation | Y | N | N | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Representative Examples | Y | Y | N | A curated set of examples are included
in tar.gz and sdist so voters can verify Hamilton works end-to-end. Not in the
whl as they serve as documentation/verification only. |
+| Other Examples | Y | N | N | These are included in the tar.gz for voters to
review but not included in the sdist or whl. |
+
+# Packages
+
+Apache Hamilton consists of 5 independently versioned packages:
+
+| Package | Key | Working Directory | Description |
+|---|---|---|---|
+| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first)
|
+| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
+| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
+| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
+| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language
server |
+
+The core `apache-hamilton` package must be released first. The other four
packages depend on it but not on each other.
+
+# Release Process
+
+## Environment Setup
+
+We recommend using [uv](https://docs.astral.sh/uv/) for Python environment
management. It handles Python versions, virtual environments, and dependency
installation in a single tool.
+
+### Prerequisites
+
+- Python 3.10+
+- `uv` ([install
guide](https://docs.astral.sh/uv/getting-started/installation/))
+- `flit` for building
+- `twine` for package validation
+- GPG key configured for signing
+- Node.js + npm for UI builds (only needed for the `ui` package)
+- Apache RAT jar for license checking (optional, for verification)
+
+```bash
+# Install uv (if not already installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a virtual environment with build dependencies
+uv venv --python 3.11
+uv pip install flit twine
+
+# Verify GPG setup
+gpg --list-secret-keys
+
+# IMPORTANT: set GPG_TTY so GPG can prompt for passphrase
+export GPG_TTY=$(tty)
+```
+
+Note: all commands below use `uv run` which automatically activates the
`.venv` environment.
+If you prefer, you can instead `source .venv/bin/activate` and omit the `uv
run` prefix.
+
+## Building a Release
+
+The main release script is `scripts/apache_release_helper.py`. It builds the
sdist and wheel, signs
+all artifacts with GPG, generates SHA512 checksums, uploads to Apache SVN, and
generates a vote email template.
+
+```bash
+# Release the core package (example: version 1.90.0, RC0)
+uv run python scripts/apache_release_helper.py --package hamilton 1.90.0 0
your_apache_id
+
+# Release a downstream package (example: sdk version 0.9.0, RC0)
+uv run python scripts/apache_release_helper.py --package sdk 0.9.0 0
your_apache_id
+```
+
+The script will:
+1. Check prerequisites (`flit`, `twine`, `gpg`)
+2. Validate the version in the source matches the version you specified
+3. Create a git tag (`apache-hamilton-v1.90.0-incubating-RC0`)
+4. Build the sdist (`.tar.gz`) and wheel (`.whl`) using `flit build
--no-use-vcs`
+5. Validate the wheel with `twine check`
+6. Sign all artifacts with GPG and generate SHA512 checksums
+7. Upload to Apache SVN dist/dev
+8. Print a vote email template
+
+Output lands in the `dist/` directory under the package's working directory.
+
+### Dry Run (no SVN upload)
+
+To test the build and signing without uploading, you can interrupt the script
after artifacts
+are built (before the SVN upload step), or comment out the upload call. The
artifacts will
+be in the `dist/` directory for inspection.
+
+### After the Vote Passes
+
+```bash
+# Push the git tag
+git push origin apache-hamilton-v1.90.0-incubating-RC0
+
+# Upload to PyPI (from the package's working directory)
+uv run twine upload dist/apache_hamilton-1.90.0.tar.gz
dist/apache_hamilton-1.90.0-py3-none-any.whl
+```
+
+# For Voters: Verifying a Release
+
+If you're voting on a release, follow these steps to verify the release
candidate.
+
+## Complete Verification Workflow
+
+```bash
+# Set version and RC number
+export VERSION=1.90.0
+export RC=0
+export PACKAGE=apache-hamilton # or apache-hamilton-sdk, etc.
+
+# 1. Download all artifacts from SVN
+svn export
https://dist.apache.org/repos/dist/dev/incubator/hamilton/${PACKAGE}-${VERSION}-incubating-RC${RC}/
hamilton-rc${RC}
+cd hamilton-rc${RC}
+
+# 2. Import KEYS file and verify GPG signatures
+wget https://downloads.apache.org/incubator/hamilton/KEYS
+gpg --import KEYS
+
+# Verify sdist signature
+gpg --verify ${PACKAGE}-${VERSION}-incubating.tar.gz.asc
${PACKAGE}-${VERSION}-incubating.tar.gz
+
+# Verify wheel signature (note: underscores in wheel filenames)
+WHEEL_NAME=$(echo ${PACKAGE} | tr '-' '_')-${VERSION}-py3-none-any.whl
+gpg --verify ${WHEEL_NAME}.asc ${WHEEL_NAME}
+
+# 3. Verify SHA512 checksums
+shasum -a 512 -c ${PACKAGE}-${VERSION}-incubating.tar.gz.sha512
+shasum -a 512 -c ${WHEEL_NAME}.sha512
+
+# 4. Extract the source archive and build from source
+tar -xzf ${PACKAGE}-${VERSION}-incubating.tar.gz
+cd ${PACKAGE}-${VERSION}-incubating/
+```
+
+## Build from Source with uv
+
+All remaining steps assume you are inside the extracted source directory
+(`${PACKAGE}-${VERSION}-incubating/`) from the step above.
+
+```bash
+# Create a fresh environment and install build tools
+uv venv --python 3.11
+uv pip install flit
+
+# Build the wheel from source
+uv run flit build --no-use-vcs
+
+# Install the wheel you just built
+uv pip install dist/apache_hamilton-${VERSION}-py3-none-any.whl
+```
+
+## Run Tests
+
+```bash
+# Install test dependencies
+uv pip install pytest pandas numpy typing_extensions typing_inspect
+
+# Run core unit tests
+uv run pytest tests/ -x -q
+
+# Run plugin tests (requires additional dependencies)
+uv pip install dask graphviz networkx pandera pydantic polars pyarrow
scikit-learn
Review Comment:
Any reason not to include this in the testing group? If there is... Make
another group called testing-plugons.
##########
scripts/apache_release_helper.py:
##########
@@ -339,29 +357,28 @@ def create_release_artifacts(package_config: dict,
version) -> list[str]:
# Use -src suffix to distinguish source distribution from wheel
(convenience package)
new_tar_ball =
f"dist/{package_name}-{version.lower()}-incubating-src.tar.gz"
_modify_tarball_for_apache_release(tarball_path[0], new_tar_ball,
package_name)
+ # Remove original flit tarball (only keep the incubating copy)
+ os.remove(tarball_path[0])
archive_name = new_tar_ball
print(f"Found source tarball: {archive_name}")
new_tar_ball_singed = sign_artifacts(archive_name)
if new_tar_ball_singed is None:
raise ValueError("Could not sign the main release artifacts.")
- # Create wheel release artifacts
+ # Wheel keeps its original PEP 427 filename (no -incubating suffix)
expected_wheel =
f"dist/{package_file_name}-{version.lower()}-py3-none-any.whl"
wheel_path = glob.glob(expected_wheel)
+ if not wheel_path:
+ raise ValueError(f"Could not find wheel: {expected_wheel}")
+ wheel_file = wheel_path[0]
- # Create incubator wheel release artifacts with -incubating suffix
- expected_incubator_wheel = (
-
f"dist/{package_name}-{version.lower()}-incubating-py3-none-any.whl"
- )
- shutil.copy(wheel_path[0], expected_incubator_wheel)
- incubator_wheel_signed_files = sign_artifacts(expected_incubator_wheel)
-
- files_to_upload = (
- [new_tar_ball]
- + new_tar_ball_singed
- + [expected_incubator_wheel]
- + incubator_wheel_signed_files
- )
+ # Verify wheel with twine before signing
+ if not verify_wheel_with_twine(wheel_file):
+ raise ValueError("Wheel failed twine validation.")
+
+ wheel_signed_files = sign_artifacts(wheel_file)
+
+ files_to_upload = [new_tar_ball] + new_tar_ball_singed + [wheel_file]
+ wheel_signed_files
Review Comment:
https://docs.astral.sh/ruff/rules/collection-literal-concatenation/
Also typo: singed -> signed
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
+
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is:
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
+
+Specifically, we set the following guidelines:
+
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | Y | Y | N | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | Y | Y | Y | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (unit + plugin) | Y | Y | N | We expect users/PMC to download the
source distribution, build from source, run the tests, and validate. Thus we
include in the tar.gz and sdist, but not in the whl. |
+| READMEs | Y | Y | Y | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions. |
+| Documentation | Y | N | N | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Representative Examples | Y | Y | N | A curated set of examples are included
in tar.gz and sdist so voters can verify Hamilton works end-to-end. Not in the
whl as they serve as documentation/verification only. |
+| Other Examples | Y | N | N | These are included in the tar.gz for voters to
review but not included in the sdist or whl. |
+
+# Packages
+
+Apache Hamilton consists of 5 independently versioned packages:
+
+| Package | Key | Working Directory | Description |
+|---|---|---|---|
+| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first)
|
+| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
+| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
+| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
+| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language
server |
+
+The core `apache-hamilton` package must be released first. The other four
packages depend on it but not on each other.
+
+# Release Process
+
+## Environment Setup
+
+We recommend using [uv](https://docs.astral.sh/uv/) for Python environment
management. It handles Python versions, virtual environments, and dependency
installation in a single tool.
+
+### Prerequisites
+
+- Python 3.10+
+- `uv` ([install
guide](https://docs.astral.sh/uv/getting-started/installation/))
+- `flit` for building
+- `twine` for package validation
+- GPG key configured for signing
+- Node.js + npm for UI builds (only needed for the `ui` package)
+- Apache RAT jar for license checking (optional, for verification)
+
+```bash
+# Install uv (if not already installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a virtual environment with build dependencies
+uv venv --python 3.11
+uv pip install flit twine
Review Comment:
Perhaps uv sync with a "build tools" group?
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
+
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is:
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
+
+Specifically, we set the following guidelines:
+
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | Y | Y | N | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | Y | Y | Y | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (unit + plugin) | Y | Y | N | We expect users/PMC to download the
source distribution, build from source, run the tests, and validate. Thus we
include in the tar.gz and sdist, but not in the whl. |
+| READMEs | Y | Y | Y | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions. |
+| Documentation | Y | N | N | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Representative Examples | Y | Y | N | A curated set of examples are included
in tar.gz and sdist so voters can verify Hamilton works end-to-end. Not in the
whl as they serve as documentation/verification only. |
+| Other Examples | Y | N | N | These are included in the tar.gz for voters to
review but not included in the sdist or whl. |
+
+# Packages
+
+Apache Hamilton consists of 5 independently versioned packages:
+
+| Package | Key | Working Directory | Description |
+|---|---|---|---|
+| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first)
|
+| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
+| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
+| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
+| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language
server |
+
+The core `apache-hamilton` package must be released first. The other four
packages depend on it but not on each other.
+
+# Release Process
+
+## Environment Setup
+
+We recommend using [uv](https://docs.astral.sh/uv/) for Python environment
management. It handles Python versions, virtual environments, and dependency
installation in a single tool.
+
+### Prerequisites
+
+- Python 3.10+
+- `uv` ([install
guide](https://docs.astral.sh/uv/getting-started/installation/))
+- `flit` for building
+- `twine` for package validation
+- GPG key configured for signing
+- Node.js + npm for UI builds (only needed for the `ui` package)
+- Apache RAT jar for license checking (optional, for verification)
+
+```bash
+# Install uv (if not already installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a virtual environment with build dependencies
+uv venv --python 3.11
+uv pip install flit twine
+
+# Verify GPG setup
+gpg --list-secret-keys
+
+# IMPORTANT: set GPG_TTY so GPG can prompt for passphrase
+export GPG_TTY=$(tty)
+```
+
+Note: all commands below use `uv run` which automatically activates the
`.venv` environment.
+If you prefer, you can instead `source .venv/bin/activate` and omit the `uv
run` prefix.
+
+## Building a Release
+
+The main release script is `scripts/apache_release_helper.py`. It builds the
sdist and wheel, signs
+all artifacts with GPG, generates SHA512 checksums, uploads to Apache SVN, and
generates a vote email template.
+
+```bash
+# Release the core package (example: version 1.90.0, RC0)
+uv run python scripts/apache_release_helper.py --package hamilton 1.90.0 0
your_apache_id
+
+# Release a downstream package (example: sdk version 0.9.0, RC0)
+uv run python scripts/apache_release_helper.py --package sdk 0.9.0 0
your_apache_id
+```
+
+The script will:
+1. Check prerequisites (`flit`, `twine`, `gpg`)
+2. Validate the version in the source matches the version you specified
+3. Create a git tag (`apache-hamilton-v1.90.0-incubating-RC0`)
+4. Build the sdist (`.tar.gz`) and wheel (`.whl`) using `flit build
--no-use-vcs`
+5. Validate the wheel with `twine check`
+6. Sign all artifacts with GPG and generate SHA512 checksums
+7. Upload to Apache SVN dist/dev
+8. Print a vote email template
+
+Output lands in the `dist/` directory under the package's working directory.
+
+### Dry Run (no SVN upload)
+
+To test the build and signing without uploading, you can interrupt the script
after artifacts
+are built (before the SVN upload step), or comment out the upload call. The
artifacts will
+be in the `dist/` directory for inspection.
+
+### After the Vote Passes
+
+```bash
+# Push the git tag
+git push origin apache-hamilton-v1.90.0-incubating-RC0
+
+# Upload to PyPI (from the package's working directory)
+uv run twine upload dist/apache_hamilton-1.90.0.tar.gz
dist/apache_hamilton-1.90.0-py3-none-any.whl
+```
+
+# For Voters: Verifying a Release
+
+If you're voting on a release, follow these steps to verify the release
candidate.
+
+## Complete Verification Workflow
+
+```bash
+# Set version and RC number
+export VERSION=1.90.0
+export RC=0
+export PACKAGE=apache-hamilton # or apache-hamilton-sdk, etc.
+
+# 1. Download all artifacts from SVN
+svn export
https://dist.apache.org/repos/dist/dev/incubator/hamilton/${PACKAGE}-${VERSION}-incubating-RC${RC}/
hamilton-rc${RC}
+cd hamilton-rc${RC}
+
+# 2. Import KEYS file and verify GPG signatures
+wget https://downloads.apache.org/incubator/hamilton/KEYS
+gpg --import KEYS
+
+# Verify sdist signature
+gpg --verify ${PACKAGE}-${VERSION}-incubating.tar.gz.asc
${PACKAGE}-${VERSION}-incubating.tar.gz
+
+# Verify wheel signature (note: underscores in wheel filenames)
+WHEEL_NAME=$(echo ${PACKAGE} | tr '-' '_')-${VERSION}-py3-none-any.whl
+gpg --verify ${WHEEL_NAME}.asc ${WHEEL_NAME}
+
+# 3. Verify SHA512 checksums
+shasum -a 512 -c ${PACKAGE}-${VERSION}-incubating.tar.gz.sha512
+shasum -a 512 -c ${WHEEL_NAME}.sha512
+
+# 4. Extract the source archive and build from source
+tar -xzf ${PACKAGE}-${VERSION}-incubating.tar.gz
+cd ${PACKAGE}-${VERSION}-incubating/
+```
+
+## Build from Source with uv
+
+All remaining steps assume you are inside the extracted source directory
+(`${PACKAGE}-${VERSION}-incubating/`) from the step above.
+
+```bash
+# Create a fresh environment and install build tools
+uv venv --python 3.11
+uv pip install flit
+
+# Build the wheel from source
+uv run flit build --no-use-vcs
+
+# Install the wheel you just built
+uv pip install dist/apache_hamilton-${VERSION}-py3-none-any.whl
+```
+
+## Run Tests
+
+```bash
+# Install test dependencies
+uv pip install pytest pandas numpy typing_extensions typing_inspect
Review Comment:
sdist includes the pyproject.toml, no? If so, I'm pretty sure this should
use uv sync with a testing group. We don't want to have have to maintain this
list separately.
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
+
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is:
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
+
+Specifically, we set the following guidelines:
+
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | Y | Y | N | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | Y | Y | Y | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (unit + plugin) | Y | Y | N | We expect users/PMC to download the
source distribution, build from source, run the tests, and validate. Thus we
include in the tar.gz and sdist, but not in the whl. |
+| READMEs | Y | Y | Y | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions. |
+| Documentation | Y | N | N | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Representative Examples | Y | Y | N | A curated set of examples are included
in tar.gz and sdist so voters can verify Hamilton works end-to-end. Not in the
whl as they serve as documentation/verification only. |
+| Other Examples | Y | N | N | These are included in the tar.gz for voters to
review but not included in the sdist or whl. |
+
+# Packages
+
+Apache Hamilton consists of 5 independently versioned packages:
+
+| Package | Key | Working Directory | Description |
+|---|---|---|---|
+| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first)
|
+| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
+| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
+| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
+| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language
server |
+
+The core `apache-hamilton` package must be released first. The other four
packages depend on it but not on each other.
+
+# Release Process
+
+## Environment Setup
+
+We recommend using [uv](https://docs.astral.sh/uv/) for Python environment
management. It handles Python versions, virtual environments, and dependency
installation in a single tool.
+
+### Prerequisites
+
+- Python 3.10+
+- `uv` ([install
guide](https://docs.astral.sh/uv/getting-started/installation/))
+- `flit` for building
+- `twine` for package validation
+- GPG key configured for signing
+- Node.js + npm for UI builds (only needed for the `ui` package)
+- Apache RAT jar for license checking (optional, for verification)
+
+```bash
+# Install uv (if not already installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a virtual environment with build dependencies
+uv venv --python 3.11
+uv pip install flit twine
+
+# Verify GPG setup
+gpg --list-secret-keys
+
+# IMPORTANT: set GPG_TTY so GPG can prompt for passphrase
+export GPG_TTY=$(tty)
+```
+
+Note: all commands below use `uv run` which automatically activates the
`.venv` environment.
+If you prefer, you can instead `source .venv/bin/activate` and omit the `uv
run` prefix.
+
+## Building a Release
+
+The main release script is `scripts/apache_release_helper.py`. It builds the
sdist and wheel, signs
+all artifacts with GPG, generates SHA512 checksums, uploads to Apache SVN, and
generates a vote email template.
+
+```bash
+# Release the core package (example: version 1.90.0, RC0)
+uv run python scripts/apache_release_helper.py --package hamilton 1.90.0 0
your_apache_id
+
+# Release a downstream package (example: sdk version 0.9.0, RC0)
+uv run python scripts/apache_release_helper.py --package sdk 0.9.0 0
your_apache_id
+```
+
+The script will:
+1. Check prerequisites (`flit`, `twine`, `gpg`)
+2. Validate the version in the source matches the version you specified
+3. Create a git tag (`apache-hamilton-v1.90.0-incubating-RC0`)
+4. Build the sdist (`.tar.gz`) and wheel (`.whl`) using `flit build
--no-use-vcs`
+5. Validate the wheel with `twine check`
+6. Sign all artifacts with GPG and generate SHA512 checksums
+7. Upload to Apache SVN dist/dev
+8. Print a vote email template
+
+Output lands in the `dist/` directory under the package's working directory.
+
+### Dry Run (no SVN upload)
+
+To test the build and signing without uploading, you can interrupt the script
after artifacts
+are built (before the SVN upload step), or comment out the upload call. The
artifacts will
+be in the `dist/` directory for inspection.
+
+### After the Vote Passes
+
+```bash
+# Push the git tag
+git push origin apache-hamilton-v1.90.0-incubating-RC0
+
+# Upload to PyPI (from the package's working directory)
+uv run twine upload dist/apache_hamilton-1.90.0.tar.gz
dist/apache_hamilton-1.90.0-py3-none-any.whl
+```
+
+# For Voters: Verifying a Release
+
+If you're voting on a release, follow these steps to verify the release
candidate.
+
+## Complete Verification Workflow
+
+```bash
+# Set version and RC number
+export VERSION=1.90.0
+export RC=0
+export PACKAGE=apache-hamilton # or apache-hamilton-sdk, etc.
+
+# 1. Download all artifacts from SVN
+svn export
https://dist.apache.org/repos/dist/dev/incubator/hamilton/${PACKAGE}-${VERSION}-incubating-RC${RC}/
hamilton-rc${RC}
+cd hamilton-rc${RC}
+
+# 2. Import KEYS file and verify GPG signatures
+wget https://downloads.apache.org/incubator/hamilton/KEYS
+gpg --import KEYS
+
+# Verify sdist signature
+gpg --verify ${PACKAGE}-${VERSION}-incubating.tar.gz.asc
${PACKAGE}-${VERSION}-incubating.tar.gz
+
+# Verify wheel signature (note: underscores in wheel filenames)
+WHEEL_NAME=$(echo ${PACKAGE} | tr '-' '_')-${VERSION}-py3-none-any.whl
+gpg --verify ${WHEEL_NAME}.asc ${WHEEL_NAME}
+
+# 3. Verify SHA512 checksums
+shasum -a 512 -c ${PACKAGE}-${VERSION}-incubating.tar.gz.sha512
+shasum -a 512 -c ${WHEEL_NAME}.sha512
+
+# 4. Extract the source archive and build from source
+tar -xzf ${PACKAGE}-${VERSION}-incubating.tar.gz
+cd ${PACKAGE}-${VERSION}-incubating/
+```
+
+## Build from Source with uv
+
+All remaining steps assume you are inside the extracted source directory
+(`${PACKAGE}-${VERSION}-incubating/`) from the step above.
+
+```bash
+# Create a fresh environment and install build tools
+uv venv --python 3.11
+uv pip install flit
+
+# Build the wheel from source
+uv run flit build --no-use-vcs
+
+# Install the wheel you just built
+uv pip install dist/apache_hamilton-${VERSION}-py3-none-any.whl
+```
+
+## Run Tests
+
+```bash
+# Install test dependencies
+uv pip install pytest pandas numpy typing_extensions typing_inspect
+
+# Run core unit tests
+uv run pytest tests/ -x -q
+
+# Run plugin tests (requires additional dependencies)
+uv pip install dask graphviz networkx pandera pydantic polars pyarrow
scikit-learn
+uv run pytest plugin_tests/ -x -q
+```
+
+## Run Examples
+
+The source archive includes representative examples to verify Hamilton works
end-to-end. Each example may require additional dependencies.
+
+### Hello World (no extra deps)
+```bash
+cd examples/hello_world
+uv run python my_script.py
Review Comment:
Should we have uv entry points for all of these? Or do we specifically want
to avoid shortcuts in this stage? Applies to all of the below as well.
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
+
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is:
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
+
+Specifically, we set the following guidelines:
+
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | Y | Y | N | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | Y | Y | Y | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (unit + plugin) | Y | Y | N | We expect users/PMC to download the
source distribution, build from source, run the tests, and validate. Thus we
include in the tar.gz and sdist, but not in the whl. |
+| READMEs | Y | Y | Y | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions. |
+| Documentation | Y | N | N | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Representative Examples | Y | Y | N | A curated set of examples are included
in tar.gz and sdist so voters can verify Hamilton works end-to-end. Not in the
whl as they serve as documentation/verification only. |
+| Other Examples | Y | N | N | These are included in the tar.gz for voters to
review but not included in the sdist or whl. |
+
+# Packages
+
+Apache Hamilton consists of 5 independently versioned packages:
+
+| Package | Key | Working Directory | Description |
+|---|---|---|---|
+| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first)
|
+| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
+| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
+| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
+| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language
server |
+
+The core `apache-hamilton` package must be released first. The other four
packages depend on it but not on each other.
+
+# Release Process
+
+## Environment Setup
+
+We recommend using [uv](https://docs.astral.sh/uv/) for Python environment
management. It handles Python versions, virtual environments, and dependency
installation in a single tool.
+
+### Prerequisites
+
+- Python 3.10+
+- `uv` ([install
guide](https://docs.astral.sh/uv/getting-started/installation/))
+- `flit` for building
+- `twine` for package validation
+- GPG key configured for signing
+- Node.js + npm for UI builds (only needed for the `ui` package)
+- Apache RAT jar for license checking (optional, for verification)
+
+```bash
+# Install uv (if not already installed)
Review Comment:
```suggestion
# Install uv (unless already installed)
```
##########
scripts/README.md:
##########
@@ -0,0 +1,290 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Policy on source versus distribution
+
+Apache Hamilton is an apache-incubating project. As such, we intend to follow
all Apache guidelines to
+both the spirit (and when applicable) the letter.
+
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is:
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
+
+Specifically, we set the following guidelines:
+
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | Y | Y | N | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | Y | Y | Y | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (unit + plugin) | Y | Y | N | We expect users/PMC to download the
source distribution, build from source, run the tests, and validate. Thus we
include in the tar.gz and sdist, but not in the whl. |
+| READMEs | Y | Y | Y | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions. |
+| Documentation | Y | N | N | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Representative Examples | Y | Y | N | A curated set of examples are included
in tar.gz and sdist so voters can verify Hamilton works end-to-end. Not in the
whl as they serve as documentation/verification only. |
+| Other Examples | Y | N | N | These are included in the tar.gz for voters to
review but not included in the sdist or whl. |
+
+# Packages
+
+Apache Hamilton consists of 5 independently versioned packages:
+
+| Package | Key | Working Directory | Description |
+|---|---|---|---|
+| `apache-hamilton` | `hamilton` | `.` | Core library (must be released first)
|
+| `apache-hamilton-sdk` | `sdk` | `ui/sdk` | Tracking SDK |
+| `apache-hamilton-contrib` | `contrib` | `contrib` | Community dataflows |
+| `apache-hamilton-ui` | `ui` | `ui/backend` | Web UI server |
+| `apache-hamilton-lsp` | `lsp` | `dev_tools/language_server` | Language
server |
+
+The core `apache-hamilton` package must be released first. The other four
packages depend on it but not on each other.
+
+# Release Process
+
+## Environment Setup
+
+We recommend using [uv](https://docs.astral.sh/uv/) for Python environment
management. It handles Python versions, virtual environments, and dependency
installation in a single tool.
+
+### Prerequisites
+
+- Python 3.10+
+- `uv` ([install
guide](https://docs.astral.sh/uv/getting-started/installation/))
+- `flit` for building
+- `twine` for package validation
+- GPG key configured for signing
+- Node.js + npm for UI builds (only needed for the `ui` package)
+- Apache RAT jar for license checking (optional, for verification)
+
+```bash
+# Install uv (if not already installed)
+curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Create a virtual environment with build dependencies
+uv venv --python 3.11
+uv pip install flit twine
+
+# Verify GPG setup
+gpg --list-secret-keys
+
+# IMPORTANT: set GPG_TTY so GPG can prompt for passphrase
+export GPG_TTY=$(tty)
+```
+
+Note: all commands below use `uv run` which automatically activates the
`.venv` environment.
+If you prefer, you can instead `source .venv/bin/activate` and omit the `uv
run` prefix.
+
+## Building a Release
+
+The main release script is `scripts/apache_release_helper.py`. It builds the
sdist and wheel, signs
+all artifacts with GPG, generates SHA512 checksums, uploads to Apache SVN, and
generates a vote email template.
+
+```bash
+# Release the core package (example: version 1.90.0, RC0)
+uv run python scripts/apache_release_helper.py --package hamilton 1.90.0 0
your_apache_id
+
+# Release a downstream package (example: sdk version 0.9.0, RC0)
+uv run python scripts/apache_release_helper.py --package sdk 0.9.0 0
your_apache_id
+```
+
+The script will:
+1. Check prerequisites (`flit`, `twine`, `gpg`)
+2. Validate the version in the source matches the version you specified
+3. Create a git tag (`apache-hamilton-v1.90.0-incubating-RC0`)
+4. Build the sdist (`.tar.gz`) and wheel (`.whl`) using `flit build
--no-use-vcs`
+5. Validate the wheel with `twine check`
+6. Sign all artifacts with GPG and generate SHA512 checksums
+7. Upload to Apache SVN dist/dev
+8. Print a vote email template
+
+Output lands in the `dist/` directory under the package's working directory.
+
+### Dry Run (no SVN upload)
+
+To test the build and signing without uploading, you can interrupt the script
after artifacts
+are built (before the SVN upload step), or comment out the upload call. The
artifacts will
+be in the `dist/` directory for inspection.
+
+### After the Vote Passes
+
+```bash
+# Push the git tag
+git push origin apache-hamilton-v1.90.0-incubating-RC0
+
+# Upload to PyPI (from the package's working directory)
+uv run twine upload dist/apache_hamilton-1.90.0.tar.gz
dist/apache_hamilton-1.90.0-py3-none-any.whl
+```
+
+# For Voters: Verifying a Release
+
+If you're voting on a release, follow these steps to verify the release
candidate.
+
+## Complete Verification Workflow
+
+```bash
+# Set version and RC number
+export VERSION=1.90.0
+export RC=0
+export PACKAGE=apache-hamilton # or apache-hamilton-sdk, etc.
+
+# 1. Download all artifacts from SVN
+svn export
https://dist.apache.org/repos/dist/dev/incubator/hamilton/${PACKAGE}-${VERSION}-incubating-RC${RC}/
hamilton-rc${RC}
+cd hamilton-rc${RC}
+
+# 2. Import KEYS file and verify GPG signatures
+wget https://downloads.apache.org/incubator/hamilton/KEYS
+gpg --import KEYS
+
+# Verify sdist signature
+gpg --verify ${PACKAGE}-${VERSION}-incubating.tar.gz.asc
${PACKAGE}-${VERSION}-incubating.tar.gz
+
+# Verify wheel signature (note: underscores in wheel filenames)
+WHEEL_NAME=$(echo ${PACKAGE} | tr '-' '_')-${VERSION}-py3-none-any.whl
+gpg --verify ${WHEEL_NAME}.asc ${WHEEL_NAME}
+
+# 3. Verify SHA512 checksums
+shasum -a 512 -c ${PACKAGE}-${VERSION}-incubating.tar.gz.sha512
+shasum -a 512 -c ${WHEEL_NAME}.sha512
+
+# 4. Extract the source archive and build from source
+tar -xzf ${PACKAGE}-${VERSION}-incubating.tar.gz
+cd ${PACKAGE}-${VERSION}-incubating/
+```
+
+## Build from Source with uv
+
+All remaining steps assume you are inside the extracted source directory
+(`${PACKAGE}-${VERSION}-incubating/`) from the step above.
+
+```bash
+# Create a fresh environment and install build tools
+uv venv --python 3.11
Review Comment:
There is a command to clean the existing venv, won't hurt to add it.
(--clean ?)
##########
scripts/verify_apache_artifacts.py:
##########
Review Comment:
I suggest running `ruff check --select=ALL --fix` on this file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
