FANNG1 opened a new pull request, #66805:
URL: https://github.com/apache/doris/pull/66805
### What problem does this PR solve?
Fixes #66772 (problem 1).
A Lance REST catalog discarded the `storage_options` a namespace vended for
a table, so any scan relying on credential vending failed:
```
open Lance dataset failed: LanceError(IO): ...
Failed to get AWS credentials: CredentialsNotLoaded("no providers in chain
provided credentials")
```
The options were re-encoded twice on the way to the BE, through one
five-entry S3-only table used in both directions:
```
namespace vends access_key_id
-> FE LanceStorageOptions.forBackend aws_access_key_id -> AWS_ACCESS_KEY
-> TFileScanRangeParams.properties
-> BE kStorageKeys AWS_ACCESS_KEY -> aws_access_key_id
-> lance-c
```
That table was written to *emit* one canonical spelling, which is correct
outbound. Reading it backwards turns it into a parser that accepts only the
spelling it happens to emit, so everything else was dropped — credentials under
any other accepted alias, and every non-S3 provider's keys, which left the
catalog unable to use credential vending outside S3 at all.
The failure was also split across the two halves, which made it hard to
read: the FE passes the vended map to the Lance Java SDK untouched, so `SHOW
TABLES` and `DESC` succeeded and only the scan failed.
The Lance Namespace specification describes `storage_options` as
configuration *"passed directly to Lance"*, so the protocol defines no key
vocabulary of its own and a client cannot assume one.
### Release note
Fixed a Lance REST catalog failing to scan with `CredentialsNotLoaded` when
it relied on credentials vended by the namespace rather than static
`s3.access_key` / `s3.secret_key`.
### What is changed and how it works?
Stop re-encoding server-supplied options.
- **`TFileScanRangeParams.lance_storage_options` (new, id 38)** carries the
options in Lance's own vocabulary. Set at ScanNode level — the same pattern as
`paimon_options` — so credentials are not serialized once per fragment split.
`properties` is deliberately not reused: it is copied wholesale into
`io::FileSystemProperties` for the shared filesystem layer.
- **FE** builds a single option map, used both by the Java SDK when it opens
the dataset and by the BE. The two can no longer disagree about how a dataset
is reached. `forBackend` is deleted.
- **BE** `_storage_options` hands that map to lance-c as it arrives;
`kStorageKeys` and the `allow_http` / `use_path_style` derivations are gone.
- The `s3()` TVF path populates the same field, including the schema-fetch
RPC.
Two details worth review attention:
**Vended aliases are renamed before merging.** `object_store` resolves an
alias and its canonical name to the same config key and keeps only one of the
two values, chosen by hash order:
```rust
// lance-io/src/object_store/providers/aws.rs
let s3_key = AmazonS3ConfigKey::from_str(&key.to_ascii_lowercase()).ok()?;
...collect() // HashMap — last write wins, iteration order randomized
```
So a namespace vending `aws_endpoint` while the catalog contributes
`endpoint` would not override it; both would survive and Lance would pick
between them unpredictably. Only the `aws_`-prefixed aliases are renamed — bare
names such as `token` mean different things across providers (S3 session token
vs. Azure bearer token) and this layer does not know the provider.
**Catalog options now use the unprefixed spelling.** `object_store` accepts
both, but the unprefixed name is also the field name used by the OpenDAL
backend, which performs no alias normalization at all.
`allow_http` is derived after merging, from whichever endpoint ends up in
use, since a namespace can replace the catalog's endpoint or supply the only
one there is.
Options that would change *which* data is read (`bucket` and its aliases,
`root`) are dropped from the vended map, mirroring the keys Lance itself
protects.
### Rolling upgrade
An FE upgraded ahead of the BEs no longer puts vended credentials into
`TFileScanRangeParams.properties`, and an older BE reads only that. A REST
catalog with no static credentials therefore cannot be scanned until the BEs
are upgraded too. The Lance catalog is not in a release yet, so this only
affects development clusters.
### How was this patch tested?
**Unit tests.** `LanceStorageOptionsTest` is new — `LanceStorageOptions` had
no test of its own, and `forBackend` had zero call sites in any test, which is
why this went unnoticed. It covers the emitted spelling, alias renaming,
`allow_http` derivation, protected keys, non-S3 options, and empty/absent
input. `LanceThriftContractTest` gains round-trip coverage of the new field.
**Regression.** The docker stub only ever vended `aws_`-prefixed keys, so
the alias path had no coverage anywhere. It now serves a second table,
`all_types_unprefixed`, backed by the same dataset but vending the unprefixed
spelling, and `test_lance_rest_catalog` scans it with no static credentials
configured.
**End to end**, against Apache Gravitino 1.3.0's `lance-rest` service, which
vends the unprefixed spelling. Catalog with no `s3.access_key` /
`s3.secret_key`:
```sql
CREATE CATALOG lance_novend PROPERTIES (
"type" = "lance",
"lance.catalog.type" = "rest",
"lance.rest.uri" = "http://127.0.0.1:9101/lance";,
"lance.namespace.parent" = "lance_catalog"
);
SELECT count(*), min(row_id), max(row_id), sum(row_id) FROM
lance_novend.doris_probe.rest_probe;
-- 1034 1 1034 535095
```
against a 1034-row, 2-fragment dataset with an IVF_FLAT index. Predicate
pushdown and vector search both return correct results on the same catalog, and
a catalog with static credentials is unaffected. Before this change the same
statement failed with `CredentialsNotLoaded`.
https://claude.ai/code/session_01M3mYXBKShBonG6Lg3br4Ld
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
