XiaoHongbo-Hope opened a new pull request, #10721: URL: https://github.com/apache/paimon/pull/10721
### Purpose A batch whose windows reference one video uses only one decoder, leaving the shared decode pool idle. Split those windows across independent cached decoders to improve LeRobot throughput for single-video datasets such as PushT. - Reuse the shared worker pool and existing cache limits; each decoder owns its stream. - Keep sample order, exact decoding and output isolation. Sort windows by frame position before splitting to reduce repeated seeks. - Add `read.video.decoders-per-payload` (default: 4; 1 disables splitting). Multi-payload batches keep their existing scheduling. No public constructor changes. Depends on #10703; the incremental implementation is commit `3c6b342118`. PushT prototype results (three paired runs, median samples/s): | Storage | Paimon baseline | Paimon, 4 decoders | Official Lance | |---|---:|---:|---:| | Local | 5,727 | 6,722 | 6,203 | | OSSFS | 5,513 | 6,699 | 5,942 | Both OSSFS stores were remounted and kernel caches cleared before each run. Eight workers, batch 32, eight-frame windows, uint8, CPU, no transforms/model; 10 warmup and 500 timed batches. These are warmed throughput results, not cache-free OSS throughput. Lance: `lerobot-lancedb` commit `40bcb659a52d`, pylance 13.0.0. Paimon used an integration branch with #10701/#10703/#10704 and startup/allocation optimizations; these are prototype measurements, not benchmarks of this PR head. On OSSFS, four decoders raised CPU use from about 8.7 to 12.2 cores and PSS from 3.32 to 3.42 GiB. Fixed-workload time was unchanged versus baseline (14.05 → 14.08 s); two decoders were faster for that short task (13.33 s). Additional decoders can increase startup cost and repeated reads. ALOHA/Koch have not been benchmarked with this change. ### Tests - 93 tests and 115 subtests passed; 29 optional-dependency tests skipped. - Real PyAV single-video output matches serial decoding. Covers duplicate/unsorted windows, uint8/float output, shared-memory isolation, decoder reuse, cache limits/eviction, pickle and existing fork/spawn cleanup. - Prototype local/OSSFS comparisons passed sample-order and separate full-tensor checks, with native reads verified in every Paimon worker. - Flake8, Python 3.6 compilation and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
