XiaoHongbo-Hope opened a new pull request, #10754: URL: https://github.com/apache/paimon/pull/10754
### Purpose LeRobot Dataset initialization reads each episode twice to validate frame ranges and collect task names. With wide Hugging Face episode metadata, this repeatedly formats unused statistics columns. Project only the required columns and iterate in batches of 1,024 rows. Preserve range validation, task mappings, and list/Pandas inputs without retaining a new metadata cache. On DROID metadata with 95,658 episodes, two Dataset constructions took **69.85 s → 2.88 s** on the same machine/environment (LeRobot 0.6.1, datasets 4.8.5, CPU). This measures initialization only; validation-context hashes match. An earlier full DataLoader experiment also confirmed that this optimization reduces startup time without improving steady-state throughput. ### Tests - LeRobot 0.4.4: `multimodal_lerobot_test.py` — 98 passed, 1 skipped. - LeRobot 0.6.1: `LeRobotValidationTest` — 70 passed, 1 skipped, 77 subtests passed. Import compatibility is covered separately by #10734. - Regression coverage: batched HF metadata, selected row order, missing controls/tasks, and list/Pandas inputs. - Flake8, Python 3.6 syntax compilation, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
