This is an automated email from the ASF dual-hosted git repository. imbajin pushed a commit to branch goal-test in repository https://gitbox.apache.org/repos/asf/hugegraph-ai.git
commit af86c0c1b64ee9af6f397decfeef7fc5d59bf7c1 Author: imbajin <[email protected]> AuthorDate: Sun May 31 01:20:52 2026 +0800 docs(quality): add quality program preflight ledger - initialize restartable campaign state and checkpoints - document current CI and service matrix scaffolds - capture PR collision and skip-control audit structure --- .workflow/quality-program/README.md | 33 ++++++++++ .../quality-program/checkpoints/00-preflight.md | 71 ++++++++++++++++++++++ .workflow/quality-program/quality-state.json | 43 +++++++++++++ .../quality-program/reports/flaky-risk-ledger.md | 8 +++ .../quality-program/reports/service-matrix.md | 7 +++ .workflow/quality-program/reports/test-matrix.md | 10 +++ .../2026-05-31-hugegraph-ai-quality-program.md | 14 ++--- 7 files changed, 179 insertions(+), 7 deletions(-) diff --git a/.workflow/quality-program/README.md b/.workflow/quality-program/README.md new file mode 100644 index 00000000..b72db668 --- /dev/null +++ b/.workflow/quality-program/README.md @@ -0,0 +1,33 @@ +# HugeGraph AI Quality Program + +This directory tracks the restartable execution state for the HugeGraph AI +Quality Program v2. + +## Source of Truth + +- `docs/superpowers/specs/2026-05-31-hugegraph-ai-quality-program-design.md` +- `docs/superpowers/plans/2026-05-31-hugegraph-ai-quality-program.md` +- `AGENTS.md` +- `hugegraph-llm/AGENTS.md` +- `rules/README.md` + +## Goal Order + +```text +P0 -> G0 -> G1 -> G2 -> G3 -> G4 -> G5 -> G6 -> G7 +``` + +## Resume Instructions + +1. Read `quality-state.json`. +2. Open the checkpoint matching `current_goal`. +3. Continue from the first unchecked step in the implementation plan. +4. Update `quality-state.json` and the matching checkpoint after every goal. + +## Hard Constraints + +- Do not silently skip selected HugeGraph integration tests. +- Use `hugegraph/hugegraph:1.7.0` for Layer B unless an exception is documented. +- Do not require real LLM, embedding, reranker, vector DB, or UI credentials outside Layer D. +- Do not perform deferred broad refactors as part of this campaign. +- Every production-code change must have a proving test and production-change-ledger entry. diff --git a/.workflow/quality-program/checkpoints/00-preflight.md b/.workflow/quality-program/checkpoints/00-preflight.md new file mode 100644 index 00000000..5c1bb482 --- /dev/null +++ b/.workflow/quality-program/checkpoints/00-preflight.md @@ -0,0 +1,71 @@ +# P0 Preflight Checkpoint + +## Repository State + +- Branch: `goal-test` +- Start SHA: `d048bc9fa07042833549dd1def5ff3222c47be70` +- Dirty files before P0 edits: none +- Workspace: Python `uv` workspace. +- Ownership: `hugegraph-llm/` is the highest-risk module; `hugegraph-python-client/` is the lower-level HugeGraph boundary dependency. +- Workflow rule: research-first, staged execution, explicit checkpoints. + +## Open PR Collision Quarantine + +Snapshot time: `2026-05-30T17:18:30Z` + +| PR | Head | Mergeable | Files | Collision surface | +|---:|---|---|---:|---| +| #350 | `yaml-config-migration` | MERGEABLE | 16 | Config migration; deferred boundary. | +| #342 | `fix/backend-metrics-strict-assertion` | MERGEABLE | 1 | Python-client targeted test fix; inspect before touching related client metrics tests. | +| #329 | `fix/number_type_vertex_id` | MERGEABLE | 7 | Python-client vertex ID behavior; quarantine for G2 ID tests/fixes. | +| #323 | `fix-gremlin-example-empty-list` | MERGEABLE | 3 | LLM Gremlin example index behavior; quarantine for G4/G5 Gremlin tests. | +| #315 | `flow_test` | MERGEABLE | 3 | Flow integration tests; avoid flow redesign and inspect before G5. | +| #277 | `yaml_config` | CONFLICTING | 5 | YAML config migration; deferred boundary. | +| #240 | `property_embedding` | CONFLICTING | 53 | Vector/property embedding and client/LLM surface; deferred boundary. | +| #222 | `auto_test_llms` | CONFLICTING | 5 | LLM auto-test/provider surface; avoid real provider credential expansion. | +| #179 | `main` | CONFLICTING | 8 | Async/streaming API refactor; deferred boundary. | +| #92 | `ragas` | CONFLICTING | 9 | RAG evaluation surface; outside default scope. | + +## Current CI Workflows + +- `.github/workflows/hugegraph-python-client.yml` + - Single `build` job for Python `3.10`, `3.11`, `3.12`. + - Starts `hugegraph/hugegraph:1.7.0` as a GitHub Actions service. + - Uses `/versions` health check with five retries. + - Runs the example script and then `uv run pytest` from `hugegraph-python-client`. +- `.github/workflows/hugegraph-llm.yml` + - Single `build` job for Python `3.10`, `3.11`. + - Starts HugeGraph manually with `docker run ... hugegraph/hugegraph:1.5.0`. + - Uses blind `sleep 10` readiness. + - Runs unit and integration paths with `SKIP_EXTERNAL_SERVICES=true`. +- `.github/workflows/ruff.yml` + - Existing style/lint gate. + +## Current Test Layout + +- Client tests live under `hugegraph-python-client/src/tests/api/`. +- LLM deterministic tests live under `hugegraph-llm/src/tests/config/`, `document/`, `middleware/`, `operators/`, `models/`, `indices/`, and `api/`. +- LLM integration tests currently include: + - `hugegraph-llm/src/tests/integration/test_graph_rag_pipeline.py` + - `hugegraph-llm/src/tests/integration/test_kg_construction.py` + - `hugegraph-llm/src/tests/integration/test_rag_pipeline.py` +- Root `pyproject.toml` has no `[tool.pytest.ini_options]` marker definitions yet. + +## Current Skip and Service Controls + +- `hugegraph-llm/src/tests/conftest.py` forces `SKIP_EXTERNAL_SERVICES=true` globally. +- LLM external tests use `SKIP_EXTERNAL_SERVICES` to skip provider/service checks. +- Client Gremlin tests support explicit `SKIP_GREMLIN_TESTS=true`. +- Client CI already uses HugeGraph `1.7.0`; LLM CI still uses `1.5.0`. +- Existing service readiness is inconsistent: client CI uses health checks, LLM CI uses `sleep 10`. + +## Abort Conditions Found + +- No dirty working-tree files before P0 edits. +- No production files edited during P0. +- Collision risk exists for client vertex ID behavior (#329), LLM Gremlin example behavior (#323), flow integration (#315), YAML config (#350/#277), vector/property embedding (#240), and async/streaming API (#179). These are quarantined for later goals. + +## Next Goal Readiness + +- Ready for G0 taxonomy and baseline work. +- First G0 edits should be limited to pytest marker definitions, test taxonomy docs, file-level markers, LLM conftest skip semantics, baseline artifacts, and checkpoint updates. diff --git a/.workflow/quality-program/quality-state.json b/.workflow/quality-program/quality-state.json new file mode 100644 index 00000000..d8de5ca4 --- /dev/null +++ b/.workflow/quality-program/quality-state.json @@ -0,0 +1,43 @@ +{ + "current_goal": "G0", + "repo_sha_start": "d048bc9fa07042833549dd1def5ff3222c47be70", + "base_branch": "main", + "open_pr_snapshot_time": "2026-05-30T17:18:30Z", + "goals_completed": [ + "P0" + ], + "files_touched": [ + ".workflow/quality-program/README.md", + ".workflow/quality-program/quality-state.json", + ".workflow/quality-program/checkpoints/00-preflight.md", + ".workflow/quality-program/reports/test-matrix.md", + ".workflow/quality-program/reports/service-matrix.md", + ".workflow/quality-program/reports/flaky-risk-ledger.md" + ], + "production_changes": [], + "tests_added_or_changed": [], + "commands_run": [ + "sed -n '1,220p' AGENTS.md", + "sed -n '1,260p' hugegraph-llm/AGENTS.md", + "sed -n '1,260p' rules/README.md", + "git status --short --branch", + "gh pr list --repo apache/hugegraph-ai --state open --limit 50 --json number,title,headRefName,baseRefName,updatedAt,mergeable,changedFiles,labels", + "sed -n '1,220p' .github/workflows/hugegraph-python-client.yml", + "sed -n '1,180p' .github/workflows/hugegraph-llm.yml", + "rg -n \"SKIP_EXTERNAL_SERVICES|SKIP_GREMLIN_TESTS|skip|xfail|pytest.mark|hugegraph:1\\\\.[0-9]|markers|addopts\" pyproject.toml hugegraph-llm/pyproject.toml hugegraph-python-client/pyproject.toml hugegraph-llm/src/tests hugegraph-python-client/src/tests .github/workflows", + "git diff --stat", + "git diff -- . ':!/.workflow/quality-program'", + "git status --short", + "git add .workflow/quality-program docs/superpowers/plans/2026-05-31-hugegraph-ai-quality-program.md", + "git commit -m \"docs(quality): add quality program preflight ledger\" -m \"- initialize restartable campaign state and checkpoints\n- document current CI and service matrix scaffolds\n- capture PR collision and skip-control audit structure\"" + ], + "known_failures": [], + "deferred_items": [ + "Avoid #350/#277 YAML config migration files unless tests require a narrow compatibility assertion.", + "Avoid #179 async/streaming API refactor surface.", + "Avoid #315 flow integration redesign surface.", + "Avoid #240 vector/property embedding abstraction surface.", + "Quarantine client/test fixes overlapping #323/#329/#342 until exact changed files are inspected." + ], + "next_recommended_action": "Start G0 baseline and test taxonomy" +} diff --git a/.workflow/quality-program/reports/flaky-risk-ledger.md b/.workflow/quality-program/reports/flaky-risk-ledger.md new file mode 100644 index 00000000..80a138fe --- /dev/null +++ b/.workflow/quality-program/reports/flaky-risk-ledger.md @@ -0,0 +1,8 @@ +# Flaky Risk Ledger + +| Test or area | Risk | Current mitigation | Future action | +|---|---|---|---| +| LLM CI HugeGraph readiness | `sleep 10` can hide slow startup or race with tests | P0 records current issue only | G1 should replace with HugeGraph `1.7.0` service health check | +| LLM external skip control | Global `SKIP_EXTERNAL_SERVICES=true` prevents selected real HugeGraph boundary tests from opting in | P0 records current issue only | G0/G1 should switch to default-only skip semantics and explicit Layer B fixture behavior | +| Client integration selection | Full client suite currently mixes service-bound and local tests | P0 records current issue only | G0/G1 should add markers and fixture-driven selection | +| Open PR collisions | Parallel PRs touch config, vertex IDs, Gremlin examples, flow tests, vector/property embedding, and async API | Quarantine list in `00-preflight.md` | Inspect exact files before goals touching those surfaces | diff --git a/.workflow/quality-program/reports/service-matrix.md b/.workflow/quality-program/reports/service-matrix.md new file mode 100644 index 00000000..8e44946c --- /dev/null +++ b/.workflow/quality-program/reports/service-matrix.md @@ -0,0 +1,7 @@ +# Service Matrix + +| Service | Default version | Used by | Health check | Required env | Failure behavior | +|---|---|---|---|---|---| +| HugeGraph Server | `hugegraph/hugegraph:1.7.0` | Layer B, Layer C graph-boundary smoke | `GET /versions` | `HUGEGRAPH_URL`, `HUGEGRAPH_REQUIRED` | fail if selected and required | +| LLM providers | explicit user/provider config only | Layer D only | provider SDK call | provider-specific API keys | excluded from default gates | +| Embedding/reranker/vector DB providers | explicit user/provider config only | Layer D only unless deterministic fake | provider-specific | provider-specific credentials | excluded from default gates | diff --git a/.workflow/quality-program/reports/test-matrix.md b/.workflow/quality-program/reports/test-matrix.md new file mode 100644 index 00000000..66d88725 --- /dev/null +++ b/.workflow/quality-program/reports/test-matrix.md @@ -0,0 +1,10 @@ +# Test Matrix + +| Layer | Module | Current command | Required services | Current issues | Target command | +|---|---|---|---|---|---| +| Layer A unit/contract | `hugegraph-python-client` | `uv run pytest` from `hugegraph-python-client` | None intended, but current suite is not split | No strict markers yet; unit and integration tests are mixed | `uv run pytest hugegraph-python-client/src/tests -m "unit or contract"` | +| Layer B HugeGraph contract | `hugegraph-python-client` | Same as full client suite | HugeGraph Server `1.7.0` in CI | Service selection not marker-driven | `HUGEGRAPH_REQUIRED=true uv run pytest hugegraph-python-client/src/tests -m "integration and hugegraph"` | +| Layer A unit/contract | `hugegraph-llm` | CI selected deterministic directories with `SKIP_EXTERNAL_SERVICES=true` | None | No strict markers yet; global conftest forces external skip | `uv run pytest hugegraph-llm/src/tests -m "unit or contract"` | +| Layer B HugeGraph boundary | `hugegraph-llm` | Integration paths run with `SKIP_EXTERNAL_SERVICES=true` | HugeGraph Server currently `1.5.0` in CI | Uses blind sleep; real boundary selection is unclear | `HUGEGRAPH_REQUIRED=true uv run pytest hugegraph-llm/src/tests -m "integration and hugegraph"` | +| Layer C core smoke | `hugegraph-llm` | Existing integration tests | Fake LLM; optional HugeGraph | Need verify production-code imports and avoid replacement pipelines | `uv run pytest hugegraph-llm/src/tests/integration -m "smoke"` | +| Layer D external | `hugegraph-llm` | Skipped by `SKIP_EXTERNAL_SERVICES=true` | Real providers or non-HugeGraph services | Should remain outside default gates | Manual or scheduled workflow with secrets | diff --git a/docs/superpowers/plans/2026-05-31-hugegraph-ai-quality-program.md b/docs/superpowers/plans/2026-05-31-hugegraph-ai-quality-program.md index 38bb1571..20c7c6d8 100644 --- a/docs/superpowers/plans/2026-05-31-hugegraph-ai-quality-program.md +++ b/docs/superpowers/plans/2026-05-31-hugegraph-ai-quality-program.md @@ -97,7 +97,7 @@ Production files may be edited only when a task says so and a test proves the be - Create: `.workflow/quality-program/reports/service-matrix.md` - Create: `.workflow/quality-program/reports/flaky-risk-ledger.md` -- [ ] **Step P0.1: Read mandatory repository guidance** +- [x] **Step P0.1: Read mandatory repository guidance** Read: @@ -109,7 +109,7 @@ sed -n '1,260p' rules/README.md Expected: confirm this is a uv workspace; client is a lower-level dependency; `hugegraph-llm` is the main high-risk module; staged workflow requires research-first and explicit checkpoints. -- [ ] **Step P0.2: Snapshot branch and open PR collision state** +- [x] **Step P0.2: Snapshot branch and open PR collision state** Run: @@ -120,7 +120,7 @@ gh pr list --repo apache/hugegraph-ai --state open --limit 50 --json number,titl Expected: capture current branch, dirty files, and open PRs. If an open PR touches the same files planned for the current goal, add it to the quarantine section of `00-preflight.md`. -- [ ] **Step P0.3: Create initial workflow state** +- [x] **Step P0.3: Create initial workflow state** Create `.workflow/quality-program/quality-state.json` with this exact shape: @@ -143,7 +143,7 @@ Create `.workflow/quality-program/quality-state.json` with this exact shape: Fill `repo_sha_start` with `git rev-parse HEAD`. Fill `open_pr_snapshot_time` with `date -u +"%Y-%m-%dT%H:%M:%SZ"`. -- [ ] **Step P0.4: Create preflight checkpoint** +- [x] **Step P0.4: Create preflight checkpoint** Create `.workflow/quality-program/checkpoints/00-preflight.md` with sections: @@ -171,7 +171,7 @@ Populate it using the commands in P0.1 and P0.2 plus: rg -n "SKIP_EXTERNAL_SERVICES|SKIP_GREMLIN_TESTS|skip|xfail|pytest.mark|hugegraph:1\\.[0-9]" hugegraph-llm/src/tests hugegraph-python-client/src/tests .github/workflows ``` -- [ ] **Step P0.5: Create test and service matrix reports** +- [x] **Step P0.5: Create test and service matrix reports** Create `.workflow/quality-program/reports/test-matrix.md` with this table header: @@ -197,7 +197,7 @@ At minimum, add HugeGraph: | HugeGraph Server | hugegraph/hugegraph:1.7.0 | Layer B, Layer C graph-boundary smoke | `GET /versions` | `HUGEGRAPH_URL`, `HUGEGRAPH_REQUIRED` | fail if selected and required | ``` -- [ ] **Step P0.6: Verify no production code changed** +- [x] **Step P0.6: Verify no production code changed** Run: @@ -208,7 +208,7 @@ git diff -- . ':!/.workflow/quality-program' Expected: only `.workflow/quality-program/*` changes exist. If production code changed, stop and revert the smallest accidental patch. -- [ ] **Step P0.7: Commit P0** +- [x] **Step P0.7: Commit P0** Run:
