yihua opened a new pull request, #845:
URL: https://github.com/apache/hudi-rs/pull/845
## Description
Upgrades `lance-core`, `lance-io`, `lance-encoding` and `lance-file` from 11
to 13 together. This replaces the four Dependabot PRs #820, #823, #824 and
#825, which can't land one at a time.
Lance 12 and later are built on `object_store` 0.14, but parquet 59 and
DataFusion 55 keep the workspace on 0.13. The Lance reader hands Lance the
store hudi-rs has already built, so this adds `lance_object_store`: an adapter
that implements the 0.14 `ObjectStore` trait over the workspace's store. Reads
are converted field by field. Writes return `NotImplemented`, since hudi-core
only reads through Lance. Lance keeps reusing hudi-rs's store and credentials,
and the adapter can be dropped along with the arrow 58 crates once the
workspace moves to `object_store` 0.14. Lance 13 is still on arrow 58, so the
`lance_arrow` bridge stays.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
The adapter has unit tests for whole-object, ranged, suffix and offset
reads, the local `File` payload, listing, `NotFound` mapping and rejected
writes. On `file://` URLs Lance reads data straight from disk and calls the
store only for size lookups. So a new reader test reads a Lance fixture through
an in-memory store, which takes the same path as s3/gs/az. It checks the batch
matches the on-disk read, and it fails if the adapter's ranged reads break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
