wangzhigang1999 opened a new pull request, #10466: URL: https://github.com/apache/paimon/pull/10466
### Purpose Fixes #10465. Reduce Python overhead when planning temporal joins with repeated group keys. `_AsOfJoinRight.plan()` currently converts each right-side metadata row to a Python tuple to find group boundaries. For one million rows across 100 groups, that means one million Python iterations to construct 100 index entries. Use Arrow `run_end_encode` on the already sorted grouping column, convert its run values and ends to Python in bulk, and populate the existing `(key,) -> (start, end)` dictionary once per group. As-of joins, linear interpolation and window joins share this code. The fast path supports a single integer, boolean, string/large-string or binary/large-binary key. Other types and composite keys retain the existing loop, as do environments without the API or the required kernel. Floating-point keys retain Python equality semantics for signed zero. Run ends use `int64` to avoid introducing a 32-bit row-count limit. Also add `.qwen` alongside the existing local tool directories in `.gitignore` to exclude local review artifacts. #### Performance End-to-end median latency against `9c3c5187a4`, with one million right-side rows and 100 groups: | Query | Storage | Before | After | Latency reduction | | --- | --- | ---: | ---: | ---: | | As-of | Local, warm cache | 1220.7 ms | 406.8 ms | 66.7% | | Linear interpolation | Local, warm cache | 1229.0 ms | 405.2 ms | 67.0% | | Window aggregation | Local, warm cache | 1245.6 ms | 411.1 ms | 67.0% | | As-of | OSS | 1788.8 ms | 979.4 ms | 45.2% | Python 3.11.13, PyArrow 19.0.1; separate baseline/candidate processes, one warm-up and three timed runs per query, four CPU cores, four Arrow CPU/I/O threads, and serial execution. Timing includes query construction, planning, reads and Python result materialization. We checked results against an independent SQLite oracle outside the timed section. The broader experiment covered 78 scenarios and 624 queries, all with correct results. Using a ±5% latency band, 66 scenarios were faster, 11 were within the band and one small-table case was slower. Additional runs did not reproduce a consistent slowdown for that case. These counts are descriptive, not a significance test. With 200,000 unique groups, peak RSS increased by 6.6–8.7 MiB. Cold reads and concurrent queries remain unmeasured. ### Tests - Full `multimodal_temporal_test.py` on Python 3.11.13 / PyArrow 19.0.1: 79 tests and 153 subtests passed. - Added result-based tests for signed-zero grouping, composite-key isolation, and fallback when the API is absent or the kernel raises `ArrowNotImplementedError`. - The same production implementation passed the full temporal suite on PyArrow 16.0.0 before the test-only cleanup. Additional baseline/candidate checks on Arrow 16 covered eight representative end-to-end scenarios and 64 queries. - Ruff and Flake8 passed. Pyright reported the same 35 existing diagnostics as the baseline, with no new diagnostics. `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
