hubgeter opened a new pull request, #67810:
URL: https://github.com/apache/doris/pull/67810
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Doris does not expose a maintenance command to compute and register Iceberg
partition statistics. Add `ALTER TABLE ... EXECUTE compute_partition_stats`,
with an optional `snapshot_id` that defaults to the current snapshot. Use the
Iceberg 1.11.0 SDK to compute or reuse a partition statistics file, register
it
in table metadata, invalidate the local cache after commit, and return its
path.
Extend `BaseExecuteAction` to support zero or multiple result rows so this
procedure can return zero rows with its column metadata intact when no
statistics
file is produced. Adapt the nine existing Iceberg actions to the new internal
return type while preserving their existing single-row schemas and values.
Include the Parquet runtime reader/writer required by the SDK. Move
Doris-generated
Parquet Thrift Java classes into a separate namespace and remove legacy
generated
sources/classes during source generation to prevent conflicts with the
official
Parquet classes. The Thrift wire definitions and generated C++ content are
unchanged.
Keep table acquisition under the existing catalog-generation fence. Carry the
catalog authenticator into the SDK's manifest workers and lazy file/stream
I/O,
which can otherwise run with an identity inherited by a shared worker thread.
The scoped table view borrows the catalog FileIO without closing it. The
action
does not add a retry around statistics writing or metadata commit, or delete
an
output file when the commit outcome is unknown.
Validation on the final revision:
- FE build and Checkstyle passed; all 57 targeted tests passed with no
failures,
errors, or skipped tests. Coverage includes result-set handling, action and
command validation, the upstream catalog-generation fence, real SDK
statistics
I/O, commit failures, and a UGI test that prewarms shared workers under
another
identity before computing statistics. Stream cleanup on authentication
failure
is covered as well.
- Started the newly built FE package with an isolated HadoopCatalog
warehouse.
Thirteen SQL cases across the nine existing actions passed, checking result
schemas/values and resulting metadata. The data-file rewrite case covered
an
empty table with no snapshot.
- Eight SQL cases for the new procedure passed: Parquet full computation,
reuse,
incremental computation and historical snapshot selection; Avro full and
incremental computation; an empty table; and a missing snapshot. Read the
actual statistics files back with the SDK and checked partition values,
record
counts, metadata registration, and unchanged data snapshot IDs. The fixture
uses data-file descriptors and real manifests/statistics, without data
scans.
- Checked class origins in the final FE package, including the single
provider
of the Iceberg Parquet runtime and the new action.
Earlier validation, before rebasing onto the catalog-authentication changes:
- Twelve fixture scenarios matched actual Spark 4.1 / Iceberg 1.11.0
procedure
calls, including statistics readback, metadata registration, and failure
cases.
- The new SQL regression suite passed through both leader and follower
endpoints;
the result file was generated by the regression framework. Follower cache
refresh after metadata registration was also verified.
- Four existing Iceberg action suites stopped at an unsupported `FORMAT_JNI`
Iceberg metadata scan in the stock BE 4.1.3 fixture; those suites did not
pass.
These Spark/follower/regression runs have not been repeated after the
rebase.
User-facing documentation is needed for the SQL syntax, snapshot selection,
result schema, and file/commit behavior. Markdown documentation is excluded
from this code PR and can be submitted separately.
### Release note
Add an Iceberg `compute_partition_stats` procedure to compute and register
partition statistics for the current or a specified snapshot.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes.
- Does this need documentation?
- [ ] No.
- [x] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
