hanahmily opened a new pull request, #1177:
URL: https://github.com/apache/skywalking-banyandb/pull/1177
## Summary
This PR introduces a vectorized pull-mode query pipeline for the trace
engine, complementing the existing push-mode path. The implementation spans
milestones M1–M7:
- **M1–M4, M6 (core pipeline)**: Two-phase synchronous pull architecture
- Phase 1: `SortedMerge` / `StaticSource` → `LimitedDistinctTraceID`
(dedup + limit + carry)
- Phase 2: `loadedCursorSource` → `GroupByTraceID` → `TraceProject` →
`TraceResult`
- `loadTraceCursorsSync` with two-gate memory budget enforcement:
hard-stop gate (cumulative ≥ budget) and metadata preflight gate (using
`uncompressedSpanSizeBytes` before `loadData` to avoid post-peak overshoot);
first-cursor exception to guarantee non-empty results
- `QueryMemoryMiB` documented and enforced as a soft span-loading
threshold (not an end-to-end cap)
- **M5**: SIDX-backed ordered query path with
`sidxInstancesToVectorizedIterators`
- **M7**: Integration into the trace `Query` dispatcher with feature-flag
gating (`--trace-vectorized-enabled`)
- **Bug fixes**:
- `LimitedDistinctTraceID` max=0 path now deduplicates via `carry.keys`
seen-check
- `GroupByTraceID` deletes bucket after emit to prevent double-emission
- Ordered SIDX path passes `maxRows=0` to `BuildMergePhase1` so
`MaxTraceSize` acts as batch granularity (matching push-path semantics) rather
than a total cap
- **Tests**:
- Parity tests: vectorized vs push path for static, SIDX lookup, SIDX
ordered, TTL, memory-quota, and over-scan cases
- Unit regression tests for each bug fix: budget preflight, dedup in max=0
path, ordered MaxTraceSize parity, duplicate-order GroupBy emission
- E2E and soak docker-compose configs extended with
`--trace-vectorized-enabled`
## Test plan
- [ ] `go test ./banyand/trace/... ./pkg/query/vectorized/trace/...` — unit
+ parity tests
- [ ] `make lint` — lint clean
- [ ] `make build` — build clean
- [ ] E2E / soak: enable `--trace-vectorized-enabled` flag in docker-compose
configs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
