raghav-reglobe opened a new pull request, #67361:
URL: https://github.com/apache/doris/pull/67361
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
`PluginDrivenMvccExternalTable.listLatestPartitions` builds a
`ListPartitionItem` per connector partition and, by design, skips the ones it
cannot build — a value that is not representable in the partition column's
type, or an arity mismatch from partition spec evolution — after which the
table degrades to UNPARTITIONED for partition pruning. Each skip is logged at
WARN **with the full stack trace, once per partition**.
Both skip shapes are table-level conditions, so they hit every partition on
every snapshot load, i.e. on every query planned against the table. The common
trigger is an Iceberg table partitioned by `(identity(flag), month(ts))`: the
connector supplies the month ordinal (e.g. `678`) keyed under the source column
`ts`, `createDateLiteral("678", datetimev2)` throws, and every partition is
skipped on every plan:
```
[PluginDrivenMvccExternalTable.listLatestPartitions():311]
toListPartitionItem failed, partitionColumns: [`flag` boolean NULL, `ts`
datetimev2(6) NULL], partitionName: flag=true/ts_month=678
org.apache.doris.common.AnalysisException: date literal [678] is invalid:
null
at
org.apache.doris.analysis.DateLiteralUtils.createDateLiteral(DateLiteralUtils.java:261)
at
org.apache.doris.analysis.LiteralExprUtils.createLiteral(LiteralExprUtils.java:72)
at
org.apache.doris.catalog.PartitionValue.getValue(PartitionValue.java:55)
at
org.apache.doris.catalog.PartitionKey.createListPartitionKeyWithTypes(PartitionKey.java:201)
at
org.apache.doris.datasource.mvcc.PluginDrivenMvccExternalTable.toListPartitionItem(PluginDrivenMvccExternalTable.java:355)
...
```
On an FE serving a few hundred such tables this is the entire warn log. In
one FE lifetime we measured 82k WARN entries / 6.0M lines (~73 stack lines
each) in `fe.warn.log`, about 1.5 GB/h across `fe.log` + `fe.warn.log` — enough
to fill the FE log volume and, on Kubernetes with an `emptyDir` log directory,
get the FE pod evicted for disk pressure.
This PR logs one WARN per listing (skipped/total, table, partition columns,
first failing partition and its cause, no stack trace) and keeps the
per-partition detail at DEBUG. No behavior change: the same partitions are
skipped and the table degrades exactly as before.
Not changed here, but probably worth a separate discussion: the LIST
fallback zips the connector's values against the *source* columns' types, which
is only meaningful for identity transforms. A
`bucket`/`truncate`/time-transform value is the transform result, not a value
of the source column, so it either fails to build (time transforms on
date/datetime columns, as above) or builds a semantically wrong list item (e.g.
a bucket ordinal typed as its INT source column). Since these items feed
`selectedPartitionNum` and the partition-count block rule rather than the read
set, this PR limits itself to the log volume.
### Release note
Reduce FE warn-log volume: an Iceberg table whose partition values cannot be
materialized as list partition items (e.g. a `month(ts)` partition spec) logged
one WARN with a stack trace per partition on every query; it now logs one
summary line per partition listing.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [x] Other reason: logging-only change; the skip/degrade path
itself is unchanged and stays covered by the existing suites that exercise
`listLatestPartitions`.
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merged this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
