This is an automated email from the ASF dual-hosted git repository.
hcr pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/mahout.git
The following commit(s) were added to refs/heads/main by this push:
new 5ef66dbb9 developement guide (#1121)
5ef66dbb9 is described below
commit 5ef66dbb94c75cc83c578a5172fdbea8494ef510
Author: Jie-Kai Chang <[email protected]>
AuthorDate: Thu Mar 5 16:16:37 2026 +0800
developement guide (#1121)
Signed-off-by: 400Ping <[email protected]>
---
CONTRIBUTING.md | 7 +-
docs/qdp/getting-started.md | 9 +-
qdp/CONTRIBUTING.md | 176 -------------------------------------
qdp/DEVELOPMENT.md | 148 +++++++++++++++++++++++++++++++
qdp/qdp-python/benchmark/README.md | 41 +++++----
5 files changed, 178 insertions(+), 203 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 459018a65..22f06829b 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -32,6 +32,8 @@ This document describes **repository-wide** setup and
workflow. For **subproject
## Quick Start
+For QDP environment and command-level setup (including root `.venv` workflow),
see [qdp/DEVELOPMENT.md](qdp/DEVELOPMENT.md).
+
### Prerequisites
- Python 3.10 (>=3.10,<3.14)
@@ -138,7 +140,7 @@ Apache Mahout includes several subprojects. Use the root
workflow above for issu
| Subproject | Guide | Description |
|------------|-------|-------------|
| **Qumat** | *(this document)* | Core Python library; use root Quick Start
and [Testing](#testing). |
-| **QDP** (Quantum Data Plane) | [qdp/CONTRIBUTING.md](qdp/CONTRIBUTING.md) |
GPU-accelerated pipeline: Rust/CUDA, DevContainer, build, install, benchmarks,
profiling. |
+| **QDP** (Quantum Data Plane) | [qdp/DEVELOPMENT.md](qdp/DEVELOPMENT.md) |
GPU-accelerated pipeline: Rust/CUDA, DevContainer, build, install, benchmarks,
profiling. |
| **Website** | [website/CONTRIBUTING.md](website/CONTRIBUTING.md) |
Docusaurus site: docs source in `/docs`, sync, versioning, deployment. |
---
@@ -146,7 +148,7 @@ Apache Mahout includes several subprojects. Use the root
workflow above for issu
## Troubleshooting
- **Pre-commit fails:** Run `pre-commit run --all-files` to see errors. Common
fixes: `cargo fmt` (Rust), `cargo clippy` (Rust lint), and ensuring you use the
repo venv (`uv run pre-commit` or `make pre-commit`).
-- **Wrong Python or missing package:** Ensure the virtual environment is
activated and you ran `uv sync --group dev` from the repo root. For QDP, see
[qdp/CONTRIBUTING.md](qdp/CONTRIBUTING.md).
+- **Wrong Python or missing package:** Ensure the virtual environment is
activated and you ran `uv sync --group dev` from the repo root. For QDP, see
[qdp/DEVELOPMENT.md](qdp/DEVELOPMENT.md).
---
@@ -155,3 +157,4 @@ Apache Mahout includes several subprojects. Use the root
workflow above for issu
- [testing/README.md](testing/README.md) — Test structure and commands
- [.github/PULL_REQUEST_TEMPLATE](.github/PULL_REQUEST_TEMPLATE) — PR
description template
- [docs/](docs/) — Documentation source (used by the website)
+- [qdp/DEVELOPMENT.md](qdp/DEVELOPMENT.md) — QDP setup, tests, benchmarks,
profiling
diff --git a/docs/qdp/getting-started.md b/docs/qdp/getting-started.md
index 33625aab0..3f9372d67 100644
--- a/docs/qdp/getting-started.md
+++ b/docs/qdp/getting-started.md
@@ -22,9 +22,10 @@ For development (from source):
```bash
git clone https://github.com/apache/mahout.git
-cd mahout/qdp/qdp-python
-uv venv -p python3.10 && source .venv/bin/activate
-uv sync --group dev && uv run maturin develop
+cd mahout
+uv sync --group dev --extra qdp
+source .venv/bin/activate
+uv run --active maturin develop --manifest-path qdp/qdp-python/Cargo.toml
```
## Quick Start
@@ -65,7 +66,7 @@ engine.encode("data.parquet", num_qubits=10,
encoding_method="amplitude") # als
| Problem | Solution |
|---------|----------|
-| Import fails | Activate venv: `source .venv/bin/activate` |
+| Import fails | Activate root venv: `source mahout/.venv/bin/activate` (or
`cd mahout && source .venv/bin/activate`) |
| CUDA errors | Run `cargo clean` in `qdp/` and rebuild |
| Out of memory | Reduce `num_qubits` or use `precision="float32"` |
diff --git a/qdp/CONTRIBUTING.md b/qdp/CONTRIBUTING.md
deleted file mode 100644
index 3a3b2913f..000000000
--- a/qdp/CONTRIBUTING.md
+++ /dev/null
@@ -1,176 +0,0 @@
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-
-# Contributing to QDP (Quantum Data Plane)
-
-This guide covers **QDP-specific** build, test, installation, and profiling.
For repository-wide workflow (issues, branches, pull requests, pre-commit), see
the root [CONTRIBUTING.md](../CONTRIBUTING.md).
-
-## Prerequisites
-
-- Linux machine (QDP currently supports Linux with NVIDIA GPUs)
-- NVIDIA GPU with CUDA driver and toolkit installed
-- Python 3.10 (>=3.10,<3.14)
-- Rust & Cargo
-
-Verify CUDA:
-
-```bash
-nvcc --version
-```
-
-## Setup Options
-
-### Option 1: Local Setup
-
-From the repository root, follow [Quick Start](../CONTRIBUTING.md#quick-start)
and install with QDP:
-
-```bash
-uv sync --group dev --extra qdp
-```
-
-### Option 2: DevContainer (Recommended)
-
-1. Open the project in VS Code.
-2. When prompted, click **Reopen in Container** (or use Command Palette: *Dev
Containers: Reopen in Container*).
-3. The container provides: CUDA toolkit, Python 3.10, Rust, development tools,
and GPU access.
-
-All commands below can be run inside the container.
-
-## Build
-
-From the **`qdp/`** directory:
-
-```bash
-cargo build -p qdp-core
-# or
-make build
-```
-
-With NVTX observability (for profiling):
-
-```bash
-make build_nvtx_profile
-```
-
-## Install as Python Package
-
-From the **repository root** you can install the main project with QDP:
-
-```bash
-uv sync --group dev --extra qdp
-```
-
-To develop the QDP Python bindings in place (e.g. for testing or benchmarks),
from **`qdp/qdp-python/`**:
-
-```bash
-uv sync --group dev
-uv run maturin develop
-```
-
-Or from **`qdp/`**:
-
-```bash
-make install
-```
-
-To install with profiling support (NVTX):
-
-```bash
-make install_profile
-```
-
-## Test
-
-### Rust unit tests
-
-From **`qdp/`**:
-
-```bash
-make test_rust
-# or
-cargo test --workspace
-```
-
-### Python tests (unified suite)
-
-From the **repository root**:
-
-```bash
-make tests
-```
-
-QDP-related tests live under `testing/qdp/` and are auto-skipped if the QDP
extension or a suitable GPU is not available. See
[testing/README.md](../testing/README.md).
-
-### Benchmarks (e2e)
-
-Benchmarks are in `qdp-python/benchmark/`. From **`qdp/`**:
-
-```bash
-make benchmark
-```
-
-This installs the package (if needed), benchmark dependencies, and runs the
benchmark suite. To run only the QDP benchmarks (no qiskit/pennylane),
uninstall those packages in your venv.
-
-Manual run from **`qdp/qdp-python/benchmark/`** (after installing the package):
-
-```bash
-python benchmark_e2e.py
-python benchmark_latency.py
-python benchmark_throughput.py
-```
-
-## Profiling and Observability
-
-### Rust examples with NVTX
-
-From **`qdp/`**:
-
-```bash
-make run_nvtx_profile # default example: nvtx_profile
-make run_nvtx_profile EXAMPLE=my_example # custom example
-```
-
-This builds the example with observability, runs it under `nsys`, and prints
profiling stats.
-
-### Python benchmarks with NVTX
-
-Install with profiling support, then run under nsys:
-
-```bash
-make install_profile
-nsys profile python qdp-python/benchmark/benchmark_e2e.py
-```
-
-See [../docs/observability/NVTX_USAGE.md](../docs/observability/NVTX_USAGE.md)
for details.
-
-## Troubleshooting
-
-| Problem | Suggestion |
-|---------|------------|
-| Python import fails after install | Use the same venv where the package was
installed; check with `python -c "import _qdp"`. Activate the venv: `source
.venv/bin/activate`. |
-| Build fails with CUDA errors | Ensure CUDA toolkit is installed and `nvcc`
is in PATH. Try `cargo clean` and rebuild. |
-| "No CUDA installed" despite having nvcc | Run `cargo clean` and build again.
|
-| Runtime: "invalid device ordinal" or "out of memory" | Check GPUs with
`nvidia-smi`, visibility with `echo $CUDA_VISIBLE_DEVICES`. Pin device:
`CUDA_VISIBLE_DEVICES=0 python your_script.py`. |
-| Benchmark failures or odd results | Install deps: `uv sync --group
benchmark` (from `qdp/qdp-python`). Check GPU memory. Uninstall
qiskit/pennylane if you only need QDP benchmarks. |
-| Pre-commit hooks fail | Run `pre-commit run --all-files`; fix formatting
with `cargo fmt`, lint with `cargo clippy`. |
-| DevContainer won’t start | Ensure Docker and NVIDIA Container Toolkit are
installed. Test: `docker run --rm --gpus all
nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi`. Rebuild without cache: *Dev
Containers: Rebuild Container Without Cache*. |
-
-## References
-
-- [qdp-python/README.md](qdp-python/README.md) — Package usage
-- [../docs/observability/NVTX_USAGE.md](../docs/observability/NVTX_USAGE.md) —
NVTX profiling
-- [../docs/test/README.md](../docs/test/README.md) — QDP test layout and
commands
diff --git a/qdp/DEVELOPMENT.md b/qdp/DEVELOPMENT.md
new file mode 100644
index 000000000..360b50fb4
--- /dev/null
+++ b/qdp/DEVELOPMENT.md
@@ -0,0 +1,148 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# QDP Development Guide
+
+This is the single practical guide for QDP contributors:
+
+- setup development environment
+- verify CUDA/GPU (`nvcc`, `nvidia-smi`)
+- build Python extension
+- run tests
+- run benchmarks
+- profile with NVTX + `nsys`
+
+## 1. Prerequisites
+
+- Linux + NVIDIA GPU
+- CUDA toolkit (must provide `nvcc`)
+- Python 3.10-3.12
+- Rust toolchain
+- `uv`
+
+Quick check:
+
+```bash
+python --version
+uv --version
+cargo --version
+nvcc --version
+nvidia-smi
+```
+
+If you meant "nccv", this guide assumes you meant `nvcc`.
+
+## 2. Unified Development Environment
+
+Use one venv at repo root (`mahout/.venv`):
+
+```bash
+cd mahout
+uv sync --group dev --extra qdp
+source .venv/bin/activate
+```
+
+Build QDP Python extension in editable mode:
+
+```bash
+uv run --active maturin develop --manifest-path qdp/qdp-python/Cargo.toml
+```
+
+Sanity check import:
+
+```bash
+uv run python -c "import _qdp; import qumat.qdp as qdp; print('QDP import ok')"
+```
+
+## 3. Development Loop
+
+Rebuild extension after Rust/PyO3 changes:
+
+```bash
+uv run --active maturin develop --manifest-path qdp/qdp-python/Cargo.toml
+```
+
+Run Rust tests:
+
+```bash
+cd qdp
+cargo test --workspace
+cd ..
+```
+
+Run Python tests:
+
+```bash
+uv run pytest testing/qdp -v
+uv run pytest qdp/qdp-python/tests -v
+```
+
+## 4. Benchmarks
+
+Install benchmark dependency group into the same root venv:
+
+```bash
+source .venv/bin/activate
+uv sync --project qdp/qdp-python --group benchmark --active
+```
+
+Run benchmark scripts:
+
+```bash
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_latency.py
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_throughput.py
+```
+
+Examples:
+
+```bash
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py --frameworks all
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_throughput.py --qubits 16 --batches 200
--batch-size 64 --prefetch 16
+```
+
+## 5. NVTX / nsys Profiling
+
+Build extension with observability feature:
+
+```bash
+uv run --active maturin develop --manifest-path qdp/qdp-python/Cargo.toml
--features observability
+```
+
+Profile a benchmark:
+
+```bash
+nsys profile --trace=cuda,nvtx --force-overwrite=true -o qdp-e2e \
+ uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py
+```
+
+Read profiling summary:
+
+```bash
+nsys stats qdp-e2e.nsys-rep
+```
+
+## 6. Common Issues
+
+- `nvcc: command not found`
+ - CUDA toolkit is not installed or not in `PATH`.
+- `No CUDA installed` during build
+ - run `cargo clean` in `qdp/`, then rebuild via `maturin develop`.
+- Import error for `_qdp`
+ - ensure you are in root `.venv` and rerun `maturin develop`.
+- Wrong GPU or OOM
+ - use `CUDA_VISIBLE_DEVICES=0` and reduce qubits / batch size.
diff --git a/qdp/qdp-python/benchmark/README.md
b/qdp/qdp-python/benchmark/README.md
index e5d50b3e2..fd1bdfe69 100644
--- a/qdp/qdp-python/benchmark/README.md
+++ b/qdp/qdp-python/benchmark/README.md
@@ -14,36 +14,37 @@ scripts:
From the repo root:
```bash
-cd qdp
-make benchmark
+uv sync --group dev --extra qdp
+source .venv/bin/activate
+uv sync --project qdp/qdp-python --group benchmark --active
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_latency.py
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_throughput.py
```
-This installs the QDP Python package (if needed), installs benchmark
-dependencies, and runs both benchmarks.
+This keeps all benchmark dependencies in the unified repo root venv
(`mahout/.venv`).
## Manual Setup
```bash
-cd qdp/qdp-python
-uv sync --group benchmark
+source .venv/bin/activate
+uv sync --project qdp/qdp-python --group benchmark --active
```
-Then run benchmarks with `uv run python ...` or activate the virtual
-environment and use `python ...`.
+Then run benchmarks with `uv run --project qdp/qdp-python python ...`.
## E2E Benchmark (Disk -> GPU)
```bash
-cd qdp/qdp-python/benchmark
-python benchmark_e2e.py
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py
```
Additional options:
```bash
-python benchmark_e2e.py --qubits 16 --samples 200 --frameworks mahout-parquet
mahout-arrow
-python benchmark_e2e.py --frameworks all
-python benchmark_e2e.py --encoding-method basis
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py --qubits 16 --samples 200
--frameworks mahout-parquet mahout-arrow
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py --frameworks all
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_e2e.py --encoding-method basis
```
Notes:
@@ -60,10 +61,9 @@ Notes:
## Data-to-State Latency Benchmark
```bash
-cd qdp/qdp-python/benchmark
-python benchmark_latency.py --qubits 16 --batches 200 --batch-size 64
--prefetch 16
-python benchmark_latency.py --frameworks mahout,pennylane
-python benchmark_latency.py --encoding-method basis
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_latency.py --qubits 16 --batches 200
--batch-size 64 --prefetch 16
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_latency.py --frameworks mahout,pennylane
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_latency.py --encoding-method basis
```
Notes:
@@ -89,10 +89,9 @@ See `qdp/qdp-python/benchmark/benchmark_throughput.md` for
details and example
output.
```bash
-cd qdp/qdp-python/benchmark
-python benchmark_throughput.py --qubits 16 --batches 200 --batch-size 64
--prefetch 16
-python benchmark_throughput.py --frameworks mahout,pennylane
-python benchmark_throughput.py --encoding-method basis
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_throughput.py --qubits 16 --batches 200
--batch-size 64 --prefetch 16
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_throughput.py --frameworks mahout,pennylane
+uv run --project qdp/qdp-python python
qdp/qdp-python/benchmark/benchmark_throughput.py --encoding-method basis
```
Notes:
