XiaoHongbo-Hope opened a new pull request, #10703: URL: https://github.com/apache/paimon/pull/10703
### Purpose LeRobot batches spanning multiple videos currently decode each payload sequentially within a camera. Decode independent payloads concurrently while preserving frame deduplication and output order. Add the table option `read.video.decode-parallelism` (default: 4; 1 for serial decoding), capped by `read.video.max-open-decoders`. This applies to video windows without `image_transforms`. The bounded pool is rebuilt in workers and drained before decoder eviction or cleanup; public Dataset interfaces remain unchanged. A diagnostic prototype with native frame lookup (#10701), using the public `lerobot/koch_pick_place_5_lego` dataset, produced these three-run medians: | Payload decode parallelism | Samples/s | Average CPU cores | Average PSS (GiB) | | --- | ---: | ---: | ---: | | 1 | 195.6 | 15.3 | 8.71 | | 4 | 294.4 | 29.9 | 9.78 | Local warm-cache CPU decoding, batch 32, 8 workers, 8-frame windows, uint8, no image transforms or model computation. Throughput improved 50.5%, using more CPU and memory; PushT showed no meaningful gain. These measurements are from the diagnostic prototype, not this PR head. ### Tests - Related Python tests: 89 passed, 101 subtests passed; 29 optional-dependency tests skipped. - Real PyAV multi-video serial/parallel pixel equality and LeRobot training readback. - Bounded concurrency, active-decoder eviction protection, failure draining, fork/spawn pool rebuilding, output isolation and Torch inference/grad modes. - Flake8, Python 3.6 syntax and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
