morningman opened a new pull request, #69488:
URL: https://github.com/apache/doris/pull/69488
### What problem does this PR solve?
Issue Number: None
Related PR: #66538, #57531, #65851
Problem Summary:
**In short.** On an Iceberg table, `DEFAULT(column)` anywhere except `UPDATE
... SET` and `MERGE ... VALUES` reads the default from the read schema, which
carries no Iceberg write-default. This PR resolves it against the write schema
that the statement pinned for its target, as branch-4.1 and branch-4.2 do.
**Background.** An Iceberg `write-default` applies to rows written after it
is set. So it lives in the writer schema that a write statement pins
(`IcebergWriteSchemaContext`), not in the columns of the read schema. #65851
resolves `DEFAULT(column)` against that pinned schema in `UPDATE ... SET`,
`MERGE ... UPDATE SET` and `MERGE ... INSERT VALUES`
(`ConnectorWriteSchemaUtils`). Every other `DEFAULT(column)` either fails at
binding or reaches `RewriteDefaultExpression` (#57531), which reads the default
from the bound column. branch-4.1 #66538, also in branch-4.2, resolves these
against the pinned write target too. It rejects `DEFAULT(column)` on an Iceberg
column in a statement that does not write the table.
**The problem, and what it cost.** Reproduced on a local cluster with a
`file://` format-version 3 table: `CREATE TABLE t (id INT, c INT)`, `ALTER
TABLE t ADD COLUMN d INT DEFAULT 42`, one row `(0, 0)`, so `d = 42`.
| Statement | Master today | After this PR (same as branch-4.2) |
|---|---|---|
| `INSERT INTO t (id, c, d) VALUES (1, 1, DEFAULT(d))` | `Unknown column 'd'
in 'table list'` | writes 42 |
| `INSERT INTO t (id, c, d) SELECT 2, 2, DEFAULT(d) FROM t WHERE id = 0` |
writes NULL | writes 42 |
| `INSERT INTO t (id, c, d) SELECT 3, 3, DEFAULT(d)` | `Unknown column 'd'`
| writes 42 |
| `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates nothing: `d = NULL`
makes the plan an empty set | updates every row, `d = 42` is pushed down to the
scan |
| `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d =
DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | matches nothing | updates
row 0 |
| `SELECT DEFAULT(d) FROM t` | NULL | `DEFAULT(d) must reference a column of
the table that this statement writes` |
The `UPDATE` and `MERGE` cases fail silently: the statement succeeds and
changes nothing.
**How this PR fixes it.**
- fe-core cannot tell from a column whether its connector keeps defaults in
the read schema. Hive does: an HMS default shows up in both schemas. Iceberg
does not. The new connector capability `SUPPORTS_WRITE_ONLY_DEFAULTS` says
which kind a table is. The Iceberg connector declares it. The Hive connector
passes it on to iceberg-on-HMS tables, like the other per-table capabilities.
- For a target that declares it, `InsertUtils.normalizePlan` resolves
`DEFAULT(column)` in `INSERT ... VALUES` and in the select lists of `INSERT ...
SELECT` against the pinned write schema. It does this by name, before the
source relations are bound, so the name always refers to a target column, as in
branch-4.2. In MySQL, `DEFAULT(column)` in a `SELECT` refers to a source column
instead.
- `RewriteDefaultExpression` takes the default for a `DEFAULT(column)` bound
to such a table from the pinned write column with the same field id. That
covers the `WHERE` of `UPDATE` and the `ON` of `MERGE`. If the statement pinned
no write schema for that table, it fails.
- Hive, Paimon and internal tables do not declare the capability, so they
behave as before.
- The new enum constant changes the connector plugin API surface, so
`connector.plugin.api.version` goes from 19.0 to 20.0, as for the other recent
changes to that surface.
**Results.**
- Local cluster, with the table above: the table shows what each statement
did before and after. `EXPLAIN UPDATE t SET c = 5 WHERE d = DEFAULT(d)` goes
from an empty set to a scan with `d = 42` pushed down to Iceberg.
- Unit tests: 3 new tests in `InsertUtilsTest` and 3 in the new
`RewriteDefaultExpressionTest`. They cover VALUES, and INSERT ... SELECT both
from a staging relation whose own column is also `name` and without FROM: each
resolves to the pinned default, and an unknown column fails. The pinned write
column wins over the read column. A statement that writes nothing fails. A
table without the capability is unchanged. With master's `InsertUtils` and
`RewriteDefaultExpression`, the 4 tests of the fix fail and the 2 tests of the
unchanged path pass. `IcebergConnectorTest`,
`HiveConnectorMetadataSiblingDelegationTest` and `ConnectorPluginSurfaceTest`
pass. So do the fe-core tests around the write schema:
`ConnectorWriteSchemaUtilsTest`, `ExternalRowLevelUpdatePlanBuilderTest`,
`ExternalRowLevelMergePlanBuilderTest`, `ChangelogRowLevelDmlTransformTest` and
`PluginDrivenExternalTableTest`.
- Regression: `test_iceberg_initial_defaults` now writes rows 20 to 22 with
`DEFAULT(default_int)` in VALUES and in INSERT ... SELECT with and without
FROM. It then updates them with `WHERE default_int = DEFAULT(default_int)` and
`MERGE ... ON ... = DEFAULT(...)`, for Parquet and ORC, and checks the rows in
the `.out`. A plain `SELECT DEFAULT(default_int)` must fail. The suite passes
locally. With master's FE it fails at the first of these statements: `Unknown
column 'default_int' in 'table list'`.
### Release note
Fix: On Iceberg tables, `DEFAULT(column)` in `INSERT ... VALUES`, `INSERT
... SELECT` and the `WHERE` / `ON` conditions of `UPDATE` and `MERGE` now
resolves to the column's write-default. Before, it failed, wrote NULL, or
matched no row. `SELECT DEFAULT(column)` on an Iceberg table now fails instead
of returning NULL.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
On a local cluster with a `file://` Iceberg catalog: `CREATE TABLE t
(id INT, c INT) PROPERTIES ('format-version'='3')`, `ALTER TABLE t ADD COLUMN d
INT DEFAULT 42`, then the statements in the table above.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
See the release note. `INSERT ... SELECT DEFAULT(column)` into an
Iceberg table names a target column, even when the source has a column of that
name.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
