skrawcz commented on code in PR #609:
URL: https://github.com/apache/burr/pull/609#discussion_r2666438725
##########
scripts/README.md:
##########
@@ -17,232 +17,109 @@
under the License.
-->
-# Burr Release Scripts
+# Policy on source versus distribution
-This directory contains helper scripts to automate the Apache release workflow.
+Apache Burr is an apache-incubating project. As such, we intend to follow all
apache guidelines to
+both the spirit (and when applicable) the letter.
-## Overview
+That said, there is occasional ambiguity. Thus we aim to clarify with a
reasonable and consistently maintained
+approach. The question that we found most ambiguous when determining our
release process is
+1. What counts as source code, and should thus be included in the "sdist" (the
source-only distribution)
+2. What should be included in the build?
-The release process has two phases:
+Specifically, we set the following guidelines:
-1. **Source-only release** (for Apache voting): Contains source code, build
scripts, and UI source—but NO pre-built artifacts
-2. **Wheel build** (for PyPI): Built from the source release, includes
pre-built UI assets
+| | source (to vote on) -- tar.gz | sdist -- source used to build | whl file |
Reasoning |
+|---|---|---|---|---|
+| Build Scripts | ✓ | ✓ | ✗ | Included in tar.gz and sdist as they are needed
to reproduce the build, but not in the whl. These are only meant to be consumed
by developers/pod members. |
+| Library Source code | ✓ | ✓ | ✓ | Core library source code is included in
all three distributions: tar.gz, sdist, and whl. |
+| Tests (integration and unit) | ✓ | ✓ | ✗ | We expect users/PMC to download
the source distribution, build from source, run the tests, and validate. Thus
we include in the tar.gz and sdist, but not in the whl. |
+| READMEs | ✓ | ✓ | ✓ | Standard project metadata files (README.md, LICENSE,
NOTICE, DISCLAIMER) are included in all three distributions: tar.gz, sdist, and
whl. |
+| Documentation | ✓ | ✗ | ✗ | Documentation source is included in the tar.gz
for voters to review, but not in the sdist or whl as it is not needed for
building or using the package. |
+| Deployment templates | ✓ | ✓ | ✓ | Convenience deployment templates are
included in tar.gz, sdist, and whl as they are referred to by specific utility
commands for deploying that are included in source. |
+| Built artifacts (UI, etc...) | ✗ | ✗ | ✓ | These are not source code and are
only included in the whl. They are created through a build process from the UI
source. Notable examples include the built npm packages. |
+| Examples (by default required for demo server) | ✓ | ✓ | ✓ | We have four
examples (see pyproject.toml) required by the demo server which can be run by a
single command. These are included in tar.gz, sdist, and whl as they are needed
for the demo functionality. |
+| Other examples | ✓ | ✗ | ✗ | These are included in the tar.gz for voters to
review but not included in the sdist or whl as they are not needed to build or
run the package. They serve more as documentation. |
-All packaging configuration lives in `pyproject.toml`:
-- `[build-system]` uses `flit_core` as the build backend
-- `[tool.flit.sdist]` controls what goes in the source tarball
-- Wheel contents are controlled by what exists in `burr/` when `flit build
--format wheel` runs
-## 1. Create the Source Release Candidate
-From the repo root:
-```bash
-python scripts/release_helper.py <version> <rc-num> <apache-id> [--dry-run]
[--build-wheel]
-```
-
-Example:
-
-```bash
-# Dry run (no git tag or SVN upload)
-python scripts/release_helper.py 0.41.0 0 myid --dry-run
-
-# Real release
-python scripts/release_helper.py 0.41.0 0 myid
-
-# With optional wheel
-python scripts/release_helper.py 0.41.0 0 myid --build-wheel
-```
-
-**What it does:**
-1. Reads version from `pyproject.toml`
-2. Cleans `dist/` directory
-3. **Removes `burr/tracking/server/build/`** to ensure no pre-built UI in
source tarball
-4. Runs `flit build --format sdist`
- - Includes files specified in `[tool.flit.sdist] include`
- - Excludes files specified in `[tool.flit.sdist] exclude`
-5. Creates Apache-branded tarball with GPG signatures and SHA512 checksums
-6. Tags git as `v{version}-incubating-RC{num}` (unless `--dry-run`)
-7. Uploads to Apache SVN (unless `--dry-run`)
-
-**Output:**
-- `dist/apache-burr-<version>-incubating.tar.gz` — ASF-branded source tarball
-- `dist/apache-burr-<version>-incubating.tar.gz.asc` — GPG signature
-- `dist/apache-burr-<version>-incubating.tar.gz.sha512` — SHA512 checksum
-
-## 2. Test the Source Release (Voter Simulation)
+# Release Process
-This simulates what Apache voters and release managers will do when validating
the release.
+**Note:** This is a work in progress and subject to change.
-**Automated testing:**
+## Environment Setup
-```bash
-bash scripts/simulate_release.sh
-```
-
-This script:
-1. Cleans `/tmp/burr-release-test/`
-2. Extracts the Apache tarball
-3. Creates a fresh virtual environment
-4. Builds UI artifacts and wheel (next step)
-5. Verifies both packages and prints their locations
-**Manual testing:**
+Prerequisites:
+- Python 3.9+
+- `flit` for building (`pip install flit`)
+- GPG key configured for signing
+- Node.js + npm for UI builds
+- Apache RAT jar for license checking (optional)
```bash
-cd /tmp
-tar -xzf /path/to/dist/apache-burr-<version>-incubating.tar.gz
-cd apache-burr-<version>-incubating
-
-# Verify source contents
-ls scripts/ # Build scripts should be present
-ls telemetry/ui/ # UI source should be present
-ls examples/ # Example directories should be present
-ls burr/tracking/server/build/ # Should NOT exist (no pre-built UI)
-
-# Create clean environment
-python -m venv venv && source venv/bin/activate
-pip install -e ".[cli]"
+# Install build dependencies
pip install flit
+pip install -e ".[cli]" # Installs burr-admin-build-ui command
-# Build artifacts and wheel (see step 3)
-python scripts/build_artifacts.py all --clean
-ls dist/*.whl
-deactivate
-```
-
-Alternatively, instead of manually creating the `venv` and installing burr
with `pip install`, you can use
-`uv` and use simplified development workflow of uv you can run the command
directly:
+# Verify GPG setup
+gpg --list-secret-keys
-```bash
-uv run scripts/build_artifacts.py all --clean
-ls dist/*.whl
+# Build UI assets (one-time or when UI changes)
+cd telemetry/ui && npm install && npm run build && cd ../..
```
-This will automatically:
+## Building Artifacts
-* download the right python version if you do not have python installed
-* create virtual environment in local `.venv` directory
-* activates the venv
-* installs `burr` in editable mode with `dev` dependency group (that contains
`cli` extra, `developer` extra
- and `flit` package.
-* deactivates the venv
+Creates the three required distributions: git archive (voting artifact), sdist
(source distribution), and wheel (binary distribution). All artifacts are
automatically signed with GPG and checksummed with SHA512. The `all` command is
the typical workflow - it builds everything in sequence.
-Next time when you run `uv run` it will also automatically sync the
environment with latest `pyproject.toml`
-
-## 3. Build Artifacts and Wheel
-
-The `build_artifacts.py` script has three subcommands:
-
-### Build everything (recommended):
+Main release script: `scripts/apache_release.py`
```bash
-python scripts/build_artifacts.py all --clean
-```
-
-This runs both `artifacts` and `wheel` subcommands in sequence.
+# Full release build (creates all artifacts, signs, checksums)
+python scripts/apache_release.py all --version 0.41.0 --rc-num 0
-### Build UI artifacts only:
+# Individual steps
+python scripts/apache_release.py archive --version 0.41.0 --rc-num 0 # Git
archive
+python scripts/apache_release.py sdist --version 0.41.0 --rc-num 0 # Source
dist
+python scripts/apache_release.py wheel --version 0.41.0 --rc-num 0 # Wheel
dist
+python scripts/apache_release.py sign --version 0.41.0 --rc-num 0 # Sign +
checksum all
-```bash
-python scripts/build_artifacts.py artifacts [--skip-install]
-```
-
-**What it does:**
-1. Checks for Node.js and npm
-2. **Cleans `burr/tracking/server/build/`** to ensure fresh UI build
-3. Installs burr from source: `pip install -e .` (unless `--skip-install`)
-4. Runs `burr-admin-build-ui`:
- - `npm install --prefix telemetry/ui`
- - `npm run build --prefix telemetry/ui`
- - **Creates `burr/tracking/server/build/`** and copies built UI into it
-5. Verifies UI assets exist in `burr/tracking/server/build/`
-
-### Build wheel only (assumes artifacts exist):
-
-```bash
-python scripts/build_artifacts.py wheel [--clean]
+# Skip UI build (if already built)
+python scripts/apache_release.py all --version 0.41.0 --rc-num 0
--skip-ui-build
```
-**What it does:**
-1. Checks for `flit`
-2. Verifies `burr/tracking/server/build/` contains UI assets
-3. Optionally cleans `dist/` (with `--clean`)
-4. Runs `flit build --format wheel`
- - **Packages all files in `burr/` directory, including
`burr/tracking/server/build/`**
- - Does NOT include files outside `burr/` (e.g., `telemetry/ui/`,
`scripts/`, `examples/`)
-5. Verifies `.whl` file was created
+Output: `dist/` directory with tar.gz (archive + sdist), whl, plus .asc and
.sha512 files.
-**Output:** `dist/apache_burr-<version>-py3-none-any.whl` (includes bundled UI)
+## Verification
-**Note:** Flit normalizes the package name `apache-burr` to `apache_burr`
(underscore) in the filename.
+Validate artifacts before uploading or voting. Checks GPG signatures, SHA512
checksums, archive integrity, and license compliance with Apache RAT. The
`list-contents` command is useful for inspecting what's actually packaged in
each artifact.
-## 4. Upload to PyPI
-
-After building the wheel:
+Verification script: `scripts/verify_apache_artifacts.py`
```bash
-twine upload dist/apache_burr-<version>-py3-none-any.whl
-```
-
-**Note:** For PyPI, you may want to publish as `burr` instead of
`apache-burr`. See the dual distribution strategy documentation.
-
-## Package Contents Reference
-
-Understanding what goes in each package type:
+# Verify signatures and checksums
+python scripts/verify_apache_artifacts.py signatures
-### Source tarball (`apache-burr-{version}-incubating.tar.gz`)
+# Verify licenses (requires Apache RAT)
+python scripts/verify_apache_artifacts.py licenses --rat-jar
/path/to/apache-rat-0.15.jar
Review Comment:
where do we get the rat jar from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
