Gabriel39 opened a new pull request, #70192:
URL: https://github.com/apache/doris/pull/70192
### What problem does this PR solve?
Related PR: #70159
The authenticated Iceberg FileIO wrapper inherits typed defaults that reject
encrypted data, delete, manifest and manifest-list descriptors before the SDK
encryption delegate can handle them. Forward all four descriptors inside the
authentication scope, preserving their key metadata and manifest-list key IDs.
This also provides the reproducible encrypted HMS metadata fixture needed
for the read roadmap:
- Disposable Moto KMS, a separate HMS and a dedicated MinIO bucket in the
standard Iceberg stack. Reference Iceberg 1.12.0 artifacts are isolated and
checksum pinned.
- Real v3 encrypted Parquet, manifests and manifest lists, with two
snapshots and distinct file key metadata. Both the SDK generic reader and Spark
SQL verify historical rows `[1, 2]` and `[1, 2, 3, 4]`.
- Actual metadata-planning failures for missing KMS, injected AWS
invalid-credential/access-denied responses, and authenticated-decryption
failure with different KMS-wrapped key material. The injected responses test
propagation; they do not claim live AWS IAM validation.
- SQL regression verifies plaintext results and 48 encrypted admission
rejections across SELECT, COUNT, DESC and system tables, caching on/off, count
pushdown on/off and repeated reads. Required fixtures fail the enabled suite if
unavailable. There is no additional test-mode switch.
The existing plugin already packages AWS KMS dependencies; this PR verifies
that closure rather than adding a second dependency. Encrypted SQL execution
remains blocked by the existing admission guard. Native encrypted file
execution is subsequent work.
### Validation
- Before the fix, all three encrypted descriptor tests failed at the
inherited FileIO defaults. After the fix, the focused 16-test suite passed.
- Full connector reactor package: 1,913 passed, 14 existing skips, zero
failures/errors.
- Full FE build and FE Checkstyle passed.
- `test_iceberg_encrypted_hms_metadata`: generated results using the
standard runner, then normal comparisons passed twice, including the final
fixture source. Each run executed 48 expected encrypted rejections and verified
plaintext rows/counts.
- Real encrypted fixture verification passed with the SDK, Spark SQL and all
four KMS/key failure checks.
### Release note
Preserve SDK encryption metadata through authenticated Iceberg FileIO.
Encrypted SQL execution remains guarded.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (full FE build and isolated external stack)
- Behavior changed:
- [x] Yes. Typed FileIO forwarding preserves SDK encryption descriptors.
- Does this need documentation?
- [x] Yes. Connector contract and fixture instructions are included.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
