wangzhaobo957-cloud opened a new pull request, #67325:
URL: https://github.com/apache/doris/pull/67325
### What problem does this PR solve?
Issue Number: close #66496
Problem Summary:
The Lance reader previously reported several Arrow and Lance-specific
types as `UNSUPPORTED`, preventing Doris from correctly discovering
schemas or reading these columns.
This PR adds Doris-side support for:
- Arrow Null as Doris `NULL`
- Arrow Duration as Doris `BIGINT`
- Arrow and Lance JSON extensions as Doris `JSON`
- Lance BFloat16 as Doris `FLOAT`
- Lance Blob v2 as Doris `VARBINARY`
The BE converts BFloat16 values to Float32 and validates the physical
storage layout of known Arrow extensions before reading them.
Only top-level Arrow Null fields are supported. Nested Null fields remain
unsupported because the current complex-type deserialization path cannot
handle them safely.
For Blob v2, this PR implements the Doris-side schema mapping and binary
column reading logic. Reading the actual Blob payload requires lance-c to
materialize the Blob descriptor as binary data. This PR does not change
the lance-c dependency.
### Release note
Add Doris-side support for Arrow Null and Duration types, and Lance JSON,
BFloat16, and Blob v2 extension types.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
Test results:
- FE unit tests: 14 passed
- BE Lance unit tests: 13 passed
- `test_lance_catalog_all_types`: passed
- `test_lance_s3_tvf`: passed
Note: The Blob v2 payload test was verified with a compatible locally
built lance-c library. The lance-c dependency is not updated by this PR.
- Behavior changed:
- [ ] No.
- [x] Yes. Lance schemas containing the supported types are now mapped
to Doris types instead of `UNSUPPORTED`.
- Does this need documentation?
- [ ] No.
- [x] Yes. A follow-up documentation PR will be submitted separately.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
