Gabriel39 opened a new pull request, #70213:
URL: https://github.com/apache/doris/pull/70213
### What problem does this PR solve?
Related PRs: #70159, #70192
Prepare the FE/BE protocol required for Iceberg encrypted reads. Data and
delete files need independent binary key/AAD payloads and explicit receiver
capabilities. Simply adding binary fields also leaks their contents through
generated Thrift strings and debug/JSON serialization.
- Add an optional versioned secret carrier: data-file field 14 and
delete-file field 12. Absence retains plaintext semantics; present
malformed/future payloads are rejected. Preserve NUL/non-UTF-8 bytes and
independent file keys. Field numbers were checked against master and all active
release branches; data-file field 13 remains reserved.
- Separate protocol recognition from executable reader capability in
heartbeats. BE reports version 1 and false reader support. FE publishes the
pair atomically, includes it in existing connector capability intersection, and
revokes it on legacy heartbeat replay/downgrade.
- Validate assigned parameters/ranges before serialization and each lazy
split response. A rejected final batch remains an error on repeated fetches and
stops its producers. BE rejects encrypted data or deletes before reader
selection, I/O/cache access or metadata COUNT fallback.
- Protect generated Java/C++ strings, nested output, ThriftDebugString, Gson
and Thrift JSON diagnostics. The narrow generation transform is wired into FE,
regression-framework and C++ generation, is idempotent, and fails on
compiler-shape drift. Binary/Compact wire encoding is retained.
- Keep encrypted dispatch blocked. Both fragment gRPC and lazy
split-response Thrift channels require authenticated, confidential transport
before keys may be sent. Reader advertisement or the Iceberg option cannot
assert this prerequisite is satisfied.
This is protocol preparation. It does not implement native decryption or
enable encrypted SQL execution. The included wire-contract document describes
field semantics, diagnostic protection, transport prerequisites and remaining
boundaries.
### Validation
- Full FE build and FE Checkstyle passed.
- New SQL regression generated its results through the standard runner, then
normal comparison passed: 80 expected rejections plus plaintext row/count
checks. Covers v2/v3, data/delete payloads, malformed/future versions, count
pushdown on/off and repeated queries. EXPLAIN proves v2 actually takes the
streaming path while current v3 planning stays synchronous. Injection is path
scoped and uses existing debug-point infrastructure; no mirrored feature/test
mode is added.
- Existing D1 admission SQL regression passed with 96 expected rejections
and plaintext comparisons.
- Final official focused FE unit run: 59 tests passed (10 protocol tests and
49 FE tests), zero failures/errors/skips. Includes a red/green regression for
terminal last-batch rejection on repeated split fetches.
- C++ standalone contract tests: 4 passed. Generator self-tests: 3 passed.
Clang-format 16, clang-tidy and BE build hygiene passed.
- Full BE ASAN unit build was attempted, but stopped in the pre-existing
`vparquet_writer_test.cpp`: its `WriteInfo` initializer uses path members
replaced by `location`. The failing test and writer headers are byte-identical
to this branch's base. Both affected BE test translation units compiled
successfully before this unrelated failure; the combined ASAN test executable
could not be linked or run. The separate four-test C++ contract run above
passed; it does not replace the full ASAN run.
SQL checks ran with the updated FE and an existing ASAN BE without
encrypted-reader advertisement. A complete mixed-version cluster test or native
encrypted SQL interoperability run is not claimed by this stage.
### Release note
Prepare versioned Iceberg encrypted scan metadata, capability checks and
redacted diagnostics. Encrypted SQL reads remain disabled.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (full FE build and local external stack)
- Behavior changed:
- [x] Yes. Explicit encrypted payloads fail closed until complete reader
and transport support is available.
- Does this need documentation?
- [x] Yes. Wire and transport contracts are included.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
