JingsongLi commented on code in PR #10734:
URL: https://github.com/apache/paimon/pull/10734#discussion_r4344714301
##########
paimon-python/pypaimon/multimodal/lerobot/writer.py:
##########
@@ -208,6 +208,15 @@ def _compute_stats(episode, features):
array = array.reshape(array.shape[0], -1)
data[name] = array
numeric_stats = compute_episode_stats(data, numeric_features)
+ # LeRobot 0.6 promotes running statistics to float32. Preserve boolean
+ # extrema from the original values to match the persisted episode schema.
+ for name, feature in numeric_features.items():
+ if feature.get("dtype") in ("bool", "boolean"):
Review Comment:
[P2] Recompute integer extrema from the original samples as well
This workaround restores boolean extrema, but LeRobot 0.6 also promotes
`int64` samples to `float64`. For a two-frame `int64` feature with shape
`(1,)`, input values `[9007199254740993, 9007199254740995]` now persist episode
min/max as `[9007199254740992, 9007199254740996]`, even though the frame values
remain exact. Values `[2**63 - 2, 2**63 - 1]` instead round up to `2**63`,
causing `flush()`/`finalize()` to fail when constructing the int64 episode
statistics columns: `ArrowInvalid: Could not convert 9.223372036854776e+18 with
type float: tried to convert to int64`.
Reproduced on this head with actual LeRobot packages, NumPy 2.2.6 and
PyArrow 22.0.0: 0.4.4 preserves both cases and commits successfully; 0.6.1
rounds the first case and fails the second. Please recompute integer
`min`/`max` from the original arrays too, before restoring multidimensional
shapes, and add large-int64 regression cases. Casting the already-rounded
extrema back to int64 would not recover the lost precision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
