This is an automated email from the ASF dual-hosted git repository.
guanmingchiu pushed a commit to branch dev-qdp
in repository https://gitbox.apache.org/repos/asf/mahout.git
The following commit(s) were added to refs/heads/dev-qdp by this push:
new 1985612e9 [QDP][Housekeeping] Clean structure and make commands (#756)
1985612e9 is described below
commit 1985612e9342df7f0c1129c7b613950fcd092860
Author: Nary Yeh <[email protected]>
AuthorDate: Thu Dec 25 18:10:16 2025 -0800
[QDP][Housekeeping] Clean structure and make commands (#756)
* refactor: benchmark/ move into qdp-python/
* build: add benchmark group in pyproject.toml
* build+docs: add make commands and update development docs
* build+docs: add profiling make command and docs
* fix: unit_test command to test_
* build: update e2e test and profile
* docs: update development docs
---
qdp/DEVELOPMENT.md | 87 +++++++++++++++++++---
qdp/Makefile | 69 +++++++++++++++++
qdp/benchmark/requirements.txt | 10 ---
qdp/qdp-python/Cargo.toml | 4 +
.../benchmark/benchmark_dataloader_throughput.py | 0
qdp/{ => qdp-python}/benchmark/benchmark_e2e.py | 0
qdp/qdp-python/pyproject.toml | 12 +++
7 files changed, 163 insertions(+), 19 deletions(-)
diff --git a/qdp/DEVELOPMENT.md b/qdp/DEVELOPMENT.md
index 251057c12..bf4465ff8 100644
--- a/qdp/DEVELOPMENT.md
+++ b/qdp/DEVELOPMENT.md
@@ -44,13 +44,55 @@ Once the container is running, you can proceed with the
build and test steps as
## Build
-Execute the following command in the `qdp/` directory to build
+Execute the following command in the `qdp/` directory to build:
```sh
cargo build -p qdp-core
```
-To build with NVTX enabled, please refer to [NVTX_USAGE
docs](./docs/observability/NVTX_USAGE.md).
+Or use the Makefile:
+
+```bash
+make build
+```
+
+To build with NVTX observability features enabled:
+
+```bash
+make build_nvtx_profile
+```
+
+## Profiling and Observability
+
+### Profiling Rust Examples
+
+To run NVTX profiling with nsys on Rust examples and view performance
statistics:
+
+```bash
+make run_nvtx_profile # Uses default nvtx_profile example
+make run_nvtx_profile EXAMPLE=my_example # Uses custom example
+```
+
+This will:
+1. Build the specified example with observability features enabled
+2. Run it with `nsys` to collect profiling data
+3. Display profiling statistics automatically
+
+### Profiling Python Benchmarks
+
+To profile Python benchmarks with NVTX annotations, you need to install the
package with profiling support:
+
+```bash
+make install_profile
+```
+
+This installs the Python package with observability features enabled. Then you
can profile any Python script:
+
+```bash
+nsys profile python qdp-python/benchmark/benchmark_e2e.py
+```
+
+For more details on NVTX profiling, markers, and how to interpret results,
please refer to [NVTX_USAGE docs](./docs/observability/NVTX_USAGE.md).
## Install as Python Package
@@ -71,31 +113,58 @@ uv sync --group dev
uv run maturin develop
```
+Alternatively, you can directly run the following command from the `qdp/`
directory:
+
+```bash
+make install
+```
+
+To install the package with profiling support (includes NVTX observability
features for performance analysis):
+
+```bash
+make install_profile
+```
+
## Test
There are two types of tests in mahout qdp: unit tests and e2e tests
(benchmark tests).
### Unit Tests
-You can simply follow the instructions in [test docs](./docs/test/README.md)
to run unit tests.
+You can use the following make commands from the `qdp/` directory:
-### E2e Tests
+```bash
+make test # Run all unit tests (Python + Rust)
+make test_python # Run Python tests only
+make test_rust # Run Rust tests only
+```
-The e2e and benchmark tests are located in the `benchmark` directory and are
written in Python. To run them, please ensure you set up the Python environment
and install the mahout qdp package following the [Install as Python
package](#install-as-python-package) section.
+Or follow the instructions in [test docs](./docs/test/README.md) to run unit
tests manually.
-Then, go to the `benchmark/` directory, where all e2e and benchmark tests are
located, and install the requirements needed for testing:
+### Benchmark Tests
-```sh
-uv pip install -r requirements.txt
+The e2e and benchmark tests are located in the `qdp-python/benchmark`
directory and are written in Python.
+
+First, ensure you set up the Python environment and install the mahout qdp
package following the [Install as Python package](#install-as-python-package)
section.
+
+To run all benchmark tests, use the make command from the `qdp/` directory:
+
+```bash
+make benchmark
```
+This will:
+1. Install the mahout qdp package if not already installed
+2. Install benchmark dependencies (`uv sync --group benchmark`)
+3. Run all benchmark tests
+
If you only want to run mahout qdp without running qiskit or pennylane
benchmark tests, simply uninstall them:
```sh
uv pip uninstall qiskit pennylane
```
-Then, run the tests:
+You can also run individual tests manually from the `qdp-python/benchmark/`
directory:
```sh
# benchmark test for dataloader throughput
diff --git a/qdp/Makefile b/qdp/Makefile
new file mode 100644
index 000000000..3d68665ae
--- /dev/null
+++ b/qdp/Makefile
@@ -0,0 +1,69 @@
+.PHONY: install install_profile install_benchmark build build_nvtx_profile
run_nvtx_profile test test_python test_rust benchmark clean help
+
+help:
+ @echo "Available targets:"
+ @echo " make install - Install the mahout python package"
+ @echo " make install_profile - Install the mahout python package
with profiling support (observability features)"
+ @echo " make install_benchmark - Install benchmark dependencies"
+ @echo " make build - Build qdp-core"
+ @echo " make build_nvtx_profile - Build qdp-core with observability
features"
+ @echo " make run_nvtx_profile - Build example, run with nsys, and
show stats (EXAMPLE=nvtx_profile)"
+ @echo " make test - Run all unit tests (Python + Rust)"
+ @echo " make test_python - Run Python unit tests only"
+ @echo " make test_rust - Run Rust unit tests only"
+ @echo " make benchmark - Run all e2e benchmark tests"
+ @echo " make clean - Clean build artifacts"
+
+install:
+ @echo "Installing mahout python package..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop
+
+install_profile:
+ @echo "Installing mahout python package with profiling support..."
+ cd qdp-python && uv sync --group dev
+ cd qdp-python && uv run maturin develop --release --features
observability
+
+build:
+ @echo "Building qdp-core..."
+ cargo build -p qdp-core
+
+build_nvtx_profile:
+ @echo "Building qdp-core with observability features..."
+ cargo build -p qdp-core --features observability --release
+
+test: test_python test_rust
+
+test_python:
+ @echo "Running Python unit tests..."
+ cd qdp-python && uv run pytest tests/
+
+test_rust:
+ @echo "Running Rust unit tests..."
+ cargo test --workspace
+
+install_benchmark:
+ cd qdp/qdp-python && uv sync --group benchmark
+
+benchmark: install install_benchmark
+ @echo "Running e2e benchmark tests..."
+ uv run python qdp-python/benchmark/benchmark_e2e.py
+ uv run python qdp-python/benchmark/benchmark_dataloader_throughput.py
+
+run_nvtx_profile:
+ $(eval EXAMPLE ?= nvtx_profile)
+ @echo "Building example '$(EXAMPLE)' with observability features..."
+ cargo build -p qdp-core --example $(EXAMPLE) --features observability
--release
+ @echo "Running '$(EXAMPLE)' with nsys profiling..."
+ nsys profile --trace=cuda,nvtx --force-overwrite=true -o report
./target/release/examples/$(EXAMPLE)
+ @echo "Showing profiling statistics..."
+ nsys stats --force-export=true report.nsys-rep
+
+clean:
+ @echo "Cleaning build artifacts..."
+ cargo clean
+ cd qdp-python && rm -rf target/
+ cd qdp-python && rm -rf .pytest_cache/
+ cd qdp-python && rm -rf __pycache__/
+ find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
+ find . -type f -name "*.pyc" -delete 2>/dev/null || true
diff --git a/qdp/benchmark/requirements.txt b/qdp/benchmark/requirements.txt
deleted file mode 100644
index 9c0ab11b2..000000000
--- a/qdp/benchmark/requirements.txt
+++ /dev/null
@@ -1,10 +0,0 @@
-numpy>=1.24,<2.0
-pandas>=2.0
-pyarrow>=14.0
-torch>=2.2
-qiskit>=1.0
-qiskit-aer>=0.17.2
-pennylane>=0.35
-scikit-learn>=1.3
-tqdm
-matplotlib
diff --git a/qdp/qdp-python/Cargo.toml b/qdp/qdp-python/Cargo.toml
index b6fbbef7e..8232f822b 100644
--- a/qdp/qdp-python/Cargo.toml
+++ b/qdp/qdp-python/Cargo.toml
@@ -10,3 +10,7 @@ crate-type = ["cdylib"]
[dependencies]
pyo3 = { version = "0.27", features = ["extension-module"] }
qdp-core = { path = "../qdp-core" }
+
+[features]
+default = []
+observability = ["qdp-core/observability"]
diff --git a/qdp/benchmark/benchmark_dataloader_throughput.py
b/qdp/qdp-python/benchmark/benchmark_dataloader_throughput.py
similarity index 100%
rename from qdp/benchmark/benchmark_dataloader_throughput.py
rename to qdp/qdp-python/benchmark/benchmark_dataloader_throughput.py
diff --git a/qdp/benchmark/benchmark_e2e.py
b/qdp/qdp-python/benchmark/benchmark_e2e.py
similarity index 100%
rename from qdp/benchmark/benchmark_e2e.py
rename to qdp/qdp-python/benchmark/benchmark_e2e.py
diff --git a/qdp/qdp-python/pyproject.toml b/qdp/qdp-python/pyproject.toml
index b4e262dd8..5f58cea2f 100644
--- a/qdp/qdp-python/pyproject.toml
+++ b/qdp/qdp-python/pyproject.toml
@@ -19,6 +19,18 @@ dev = [
"pytest>=9.0.1",
"torch>=2.2",
]
+benchmark = [
+ "numpy>=1.24,<2.0",
+ "pandas>=2.0",
+ "pyarrow>=14.0",
+ "torch>=2.2",
+ "qiskit>=1.0",
+ "qiskit-aer>=0.17.2",
+ "pennylane>=0.35",
+ "scikit-learn>=1.3",
+ "tqdm",
+ "matplotlib",
+]
[[tool.uv.index]]
name = "pytorch"
