FANNG1 opened a new issue, #13338: URL: https://github.com/apache/gravitino/issues/13338
### Describe the feature Lance has two blob column representations: - **Legacy blob**: an Arrow `LargeBinary` field carrying field metadata `lance-encoding:blob = "true"`. Rejected by Lance for file version >= 2.2. - **Blob v2**: `Struct<data: LargeBinary?, uri: Utf8?>` (writers also accept trailing `position: UInt64?`, `size: UInt64?`), tagged with the Arrow extension metadata `ARROW:extension:name = "lance.blob.v2"`, plus optional `lance-encoding:blob-inline-size-threshold` / `lance-encoding:blob-dedicated-size-threshold`. Neither is modeled by Gravitino, and **blob v2 loses information on the round trip**. ### Motivation `LanceDataTypeConverter` (`lance/lance-common/.../LanceDataTypeConverter.java`) handles the two forms differently, and both outcomes are bad: 1. **Legacy blob** — Arrow `LargeBinary` has no case in `toGravitino` (only `Binary` -> `Types.BinaryType`), so it falls through to the catch-all and becomes `Types.ExternalType` holding the serialized Arrow field JSON. Lossless, because Arrow's `Field` serializes its metadata, but completely opaque to Gravitino, its clients and the web UI. 2. **Blob v2** — Arrow `Struct` *does* match `case Struct:`, so the field is converted to a plain `Types.StructType` and **the field metadata is silently dropped**. The reverse direction cannot restore it either: `toArrowField`'s `STRUCT` branch builds `new FieldType(nullable, ArrowType.Struct.INSTANCE, null)` with null metadata. A blob column loaded through Gravitino comes back as an ordinary struct, and nothing downstream can tell it was a blob. Beyond fidelity: users cannot create a blob column through Gravitino's `createTable` at all, except by hand-writing the raw Arrow JSON into an `ExternalType`. ### Describe the solution Two steps, the first of which is the correctness fix: 1. **Stop dropping Arrow field metadata.** In `toGravitino`, when a field carries metadata that the target Gravitino type cannot express, fall back to `Types.ExternalType` instead of the native mapping. This makes blob v2 (and any other Arrow extension type) round-trip losslessly, since the `ExternalType` JSON already carries `metadata`. 2. **Model blobs natively**, e.g. a `Types.BlobType`, mapped in both directions to the legacy `LargeBinary` + `lance-encoding:blob` form and to the v2 extension-typed struct, so blob columns are creatable and readable as a real type from the REST API, the Java/Python clients and the web UI. Storage-layout hints (`inline-size-threshold`, `dedicated-size-threshold`) can be carried as column properties. ### Additional context - Verified against `lance-core` 6.0.0 (`gradle/libs.versions.toml:34`) and Lance source: `rust/lance-arrow/src/lib.rs` (`BLOB_V2_EXT_NAME`), `rust/lance/src/blob.rs` (`blob_field`). - Companion to #13337, which proposes a native vector type; both stem from the same `ExternalType` catch-all in `LanceDataTypeConverter`. - If step 1 is considered a bug rather than a feature, it can be split into its own bug report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
