XiaoHongbo-Hope opened a new pull request, #10483:
URL: https://github.com/apache/paimon/pull/10483
### Purpose
Allow LeRobot v3 imports to run across Ray workers instead of a single
process:
```python
conn.load_from_lerobot(
"robot_data", "oss://bucket/lerobot_dataset",
engine="ray", concurrency=8, source_options=source_options,
)
```
Workers reuse the existing validation/writer path and prepare complete
Episode groups. Episodes sharing video files stay together. The driver commits
frames only after every task succeeds, then publishes companion metadata, the
initial BTree and optional tags. Existing calls remain single-process.
Requires an initialized Ray cluster and shared source/target storage.
Supports FileIO URIs and shared directories; Hub IDs remain on the
single-process path. Tasks are not automatically retried. Publication across
tables is not atomic; failed imports may leave partial tables and uncommitted
files.
### Tests
72 passed, 29 optional-dependency skips. Six new tests cover grouping, API
validation, real Ray-worker image/video imports against serial results, tags
and worker failure without a frame commit. Flake8 and `git diff --check` passed.
Validated on a local two-CPU Ray cluster (Ray 2.55.1); multi-node OSS
performance has not been measured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
