YannByron opened a new pull request, #10567: URL: https://github.com/apache/paimon/pull/10567
### Purpose CWD currently discards the scan plan used to build its window index and replans the snapshot for each payload projection and worker. For task-restricted queries, this repeats planning beyond the context already selected by the query. This change plans and authorizes index and payload columns together, reads only group/order/row-ID columns into the index, and reuses the resulting split metadata across payload projections and worker serialization. Indexed split ranges already narrow row reads, so the redundant row-ID predicate is removed while retaining the precise row-ID filter before BLOB body reads. The optional `anchor_row_ids` argument selects eligible window anchors without removing their history or future context. It preserves the existing group/order schedule, stride, boundaries and requested sample order. The context query must still include complete groups when original episode boundaries are required. This does not add externally supplied split assignments, a custom sampler, or checkpoint recovery. ### Tests - Affected CWD, planning, Torch reader, window transform, multimodal table, ACT benchmark and query-authorization suites: **290 passed, 5 skipped** (Ray is not installed). - Native planner/reader CWD lane: **51 passed**, exercising 62 native plans and 161 native reads. - Three regression tests fail against upstream `937b5e1d51a5d346634df56794c562150aa1a2b5` and pass with this change: repeated context planning after projection/pickle, redundant predicate filtering, and projection-sensitive payload authorization. - Additional coverage checks sparse/empty anchors, stride and boundary behavior, duplicate and reordered requests, independent data-evolution column files, nonzero file offsets, task pruning and serialization. Flake8 and the Python license checker pass. A reproducible CPU benchmark is included. The measurements below compare the same immutable snapshots and sample sequence against the upstream commit above. Each case uses one discarded warm-up pair and five measured pairs, alternating baseline/candidate order in fresh processes; all sample and decoded-image digests match. Results are medians on an Apple M2 Pro, without DataLoader workers; OS caches are uncontrolled. | Workload | Metric | Baseline | This change | Change | | --- | --- | ---: | ---: | ---: | | Synthetic, single task | First batch | 75.10 ms | 10.13 ms | -86.5% | | Synthetic, single task | Steady read throughput | 1,460 samples/s | 1,825 samples/s | +25.0% | | Synthetic, all tasks | Steady read throughput | 468 samples/s | 548 samples/s | +17.2% | | Real images, one episode | Read + decode throughput | 352 samples/s | 350 samples/s | -0.8% | | Real images, four episodes | Read + decode throughput | 335 samples/s | 341 samples/s | +1.7% | The synthetic fixture has 16 tasks, 8 episodes per task, 128 frames per episode and 1 KB BLOBs. The real-image probe reads 144 samples per run from a fixed four-episode snapshot, with three RGB views and 32 action steps. Real-image run ranges overlap; decoded throughput is effectively unchanged. These measurements exclude model/optimizer execution, augmentation and resizing, and do not establish an end-to-end improvement over HDF5 or LeRobot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
