XiaoHongbo-Hope opened a new pull request, #10732: URL: https://github.com/apache/paimon/pull/10732
### Purpose Native LeRobot readers still pickle the Python fallback splits into each spawned worker, even when those splits are never used. Omit them when native planning or reading is enabled, and rebuild them from the pinned snapshot only if Python fallback is needed. Python-only readers retain their splits and fallback scanners remain reusable across batches. This addresses a worker-startup bottleneck found in the public Koch LeRobot benchmark: first-batch time fell from about 24.78 s to 4.87 s in the earlier local experiment (8 forkserver workers, batch 32). These are experiment results, not a fresh benchmark of this PR head. ### Tests - Native reader suite: 17 passed, covering spawn, pickle, pinned snapshots, both native options, fallback scanner reuse, and Python-only reads. - Repository flake8, Python 3.6 syntax compilation, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
