FANNG1 commented on PR #13929:
URL: https://github.com/apache/gravitino/pull/13929#issuecomment-6648551140
Thanks for the review — and for #14556. Rebasing onto it removed two
findings I was about to file separately: the null-body asymmetry on the alter
paths, and malformed bodies bypassing `LanceExceptionMapper`.
**On the write regression: I was wrong, and the change you suggested would
not have helped.** I measured it instead of reasoning from the defaults.
`lance-spark` 0.2.0 never reads the target dataset's file format version.
`LanceDataWriter#buildWriteParams` sets one only from the Spark option, and the
write goes through `Fragment.create(...)` →
`FragmentCreateBuilder::write_fragments_v2_impl`, which resolves it as
`params.data_storage_version.unwrap_or_default()`
(`v2.0.0:rust/lance/src/dataset/fragment/write.rs:216`) — the manifest is never
consulted. At `lance-core` 2.0.0 that default is `V2_0`, so every write from
that bundle emits a 2.0 data file. Meanwhile `createDataset` leaves the version
unset when `requiredFileFormatVersion` is empty, so the dataset takes
`DataStorageFormat::default()`: **2.1** on `lance-core` 6.0.0, **2.2** on
12.0.0. `check_storage_version` compares the two strictly.
With the skip removed:
| dataset | emulates | `INSERT` from 0.2.0 |
|---|---|---|
| empty, pinned 2.1 | pre-upgrade `lance-core` 6.0.0 | refused — `The
operation added files with version 2.0. However, the data storage version is
2.1.` (`commit.rs:365`) |
| seeded 2 rows, 2.2 | this PR | refused — `All data files must have the
same version. Detected both 2.2 and 2.0` (`fragment.rs:454`) |
I built the 2.1 arm by re-pinning the freshly created empty dataset with
`OVERWRITE` + `dataStorageVersion("2.1")`, which is equivalent to what 6.0.0
wrote. So this upgrade changes only the number in the message — and improves
it, since the pre-upgrade failure was an `Error::Internal` ("Encountered
internal error. Please file a bug report") and the current one is a proper
`invalid_input`.
That is why pinning the old default when `requiredFileFormatVersion` is
empty would not help: it reproduces exactly the state that still refuses the
write. Only **2.0** would let the bundle write, and that costs every user 2.1
and 2.2 — `lance.blob` requires 2.2, so blob v2 tables would become impossible.
I'd rather not trade that for one bundle three releases behind, but say so if
you read it differently.
The fix is client-side and already shipped: `lance-spark` 0.4.0 reads the
dataset's version in `LanceDataset#newWriteBuilder`. So no issue to file, and
the IT comment, `docs/lakehouse-generic-lance-table.md` and the PR description
no longer call it a regression.
**One correction I owe:** the description and the selector commit message
said a payload carrying `branch` used to fail with a `500`. It was a `400` with
a raw Jackson message as `text/plain`, from `JacksonFeature`'s own
`JsonMappingExceptionMapper`, which outranks `LanceExceptionMapper` because the
latter is an `ExceptionMapper<Throwable>`. The 500 I saw came from the test
harness, which registers neither. Commit message reworded, description fixed.
Rewriting the test also showed it was blind to the upgrade: every test in
`LanceSparkRESTServiceIT` passed with `lance-core` reverted to 6.0.0, because
the only version-sensitive thing in the file was a skip list. The read test now
asserts `Dataset#getLanceFileFormatVersion` is `2.2`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]