yihua opened a new pull request, #989:
URL: https://github.com/apache/hudi-rs/pull/989
## Description
closes #988
A table open, plan and read used to build a new object store (HTTP client,
connection pool, credential provider) at every `Storage::new` call. Now the
table keeps the `Storage` it opened with, and derives the timeline, file system
view, file group readers and the metadata table from it through a new
`Storage::derive`. A derived `Storage` reuses the parent's store when the
options are equal and the base path has the same store root: scheme and
authority, plus the bucket or container that `object_store` takes from the
path. It keeps its own base URL, so the object keys it reads don't change. A
different bucket or different options still builds its own store, and nothing
is shared across tables, because a store holds resolved credentials. The
standalone `FileGroupReader::new_with_options` used by the bindings also builds
one store instead of two.
One case still builds a second store: the metadata table is opened without
the data table's storage options, so it shares only when the user passed none,
as with credentials from env vars, profiles or instance metadata. That is
unchanged by this PR and worth its own fix.
## How are the changes test-covered
- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
- [ ] Details are described below
`table_open_plan_and_read_build_one_object_store` opens a MOR table with a
metadata table, plans, reads and computes stats, and asserts that exactly one
store is built (it counts 9 on main). It also asserts that the metadata table
actually opened. `new_with_options_builds_one_object_store` covers the
standalone reader. The storage unit tests cover sharing under the same root and
options, a new store for another bucket, scheme or option value, Azure
containers carried in the path, and reads through a derived `Storage` at its
own base path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]