FANNG1 commented on PR #13929:
URL: https://github.com/apache/gravitino/pull/13929#issuecomment-6648551140

   Thanks for the review — and for #14556. Rebasing onto it removed two 
findings I was about to file separately: the null-body asymmetry on the alter 
paths, and malformed bodies bypassing `LanceExceptionMapper`.
   
   **On the write regression: I was wrong, and the change you suggested would 
not have helped.** I measured it instead of reasoning from the defaults.
   
   `lance-spark` 0.2.0 never reads the target dataset's file format version. 
`LanceDataWriter#buildWriteParams` sets one only from the Spark option, and the 
write goes through `Fragment.create(...)` → 
`FragmentCreateBuilder::write_fragments_v2_impl`, which resolves it as 
`params.data_storage_version.unwrap_or_default()` 
(`v2.0.0:rust/lance/src/dataset/fragment/write.rs:216`) — the manifest is never 
consulted. At `lance-core` 2.0.0 that default is `V2_0`, so every write from 
that bundle emits a 2.0 data file. Meanwhile `createDataset` leaves the version 
unset when `requiredFileFormatVersion` is empty, so the dataset takes 
`DataStorageFormat::default()`: **2.1** on `lance-core` 6.0.0, **2.2** on 
12.0.0. `check_storage_version` compares the two strictly.
   
   With the skip removed:
   
   | dataset | emulates | `INSERT` from 0.2.0 |
   |---|---|---|
   | empty, pinned 2.1 | pre-upgrade `lance-core` 6.0.0 | refused — `The 
operation added files with version 2.0.  However, the data storage version is 
2.1.` (`commit.rs:365`) |
   | seeded 2 rows, 2.2 | this PR | refused — `All data files must have the 
same version.  Detected both 2.2 and 2.0` (`fragment.rs:454`) |
   
   I built the 2.1 arm by re-pinning the freshly created empty dataset with 
`OVERWRITE` + `dataStorageVersion("2.1")`, which is equivalent to what 6.0.0 
wrote. So this upgrade changes only the number in the message — and improves 
it, since the pre-upgrade failure was an `Error::Internal` ("Encountered 
internal error. Please file a bug report") and the current one is a proper 
`invalid_input`.
   
   That is why pinning the old default when `requiredFileFormatVersion` is 
empty would not help: it reproduces exactly the state that still refuses the 
write. Only **2.0** would let the bundle write, and that costs every user 2.1 
and 2.2 — `lance.blob` requires 2.2, so blob v2 tables would become impossible. 
I'd rather not trade that for one bundle three releases behind, but say so if 
you read it differently.
   
   The fix is client-side and already shipped: `lance-spark` 0.4.0 reads the 
dataset's version in `LanceDataset#newWriteBuilder`. So no issue to file, and 
the IT comment, `docs/lakehouse-generic-lance-table.md` and the PR description 
no longer call it a regression.
   
   **One correction I owe:** the description and the selector commit message 
said a payload carrying `branch` used to fail with a `500`. It was a `400` with 
a raw Jackson message as `text/plain`, from `JacksonFeature`'s own 
`JsonMappingExceptionMapper`, which outranks `LanceExceptionMapper` because the 
latter is an `ExceptionMapper<Throwable>`. The 500 I saw came from the test 
harness, which registers neither. Commit message reworded, description fixed.
   
   Rewriting the test also showed it was blind to the upgrade: every test in 
`LanceSparkRESTServiceIT` passed with `lance-core` reverted to 6.0.0, because 
the only version-sensitive thing in the file was a skip list. The read test now 
asserts `Dataset#getLanceFileFormatVersion` is `2.2`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to