This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 7e6f6cb3a34 [fix](fe) Resolve DEFAULT(column) on Iceberg tables from
the write schema (#69488)
7e6f6cb3a34 is described below
commit 7e6f6cb3a34ab11ad0399cb10219c18f96763a93
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Nov 2 09:47:01 2026 +0000
[fix](fe) Resolve DEFAULT(column) on Iceberg tables from the write schema
(#69488)
### What problem does this PR solve?
Issue Number: None
Related PR: #66538, #57531, #65851
Problem Summary:
**In short.** On an Iceberg table, `DEFAULT(column)` anywhere except
`UPDATE ... SET` and `MERGE ... VALUES` reads the default from the read
schema, which carries no Iceberg write-default. This PR resolves it
against the write schema that the statement pinned for its target, as
branch-4.1 and branch-4.2 do.
**Background.** An Iceberg `write-default` applies to rows written after
it is set. So it lives in the writer schema that a write statement pins
(`IcebergWriteSchemaContext`), not in the columns of the read schema.
#65851 resolves `DEFAULT(column)` against that pinned schema in `UPDATE
... SET`, `MERGE ... UPDATE SET` and `MERGE ... INSERT VALUES`
(`ConnectorWriteSchemaUtils`). Every other `DEFAULT(column)` either
fails at binding or reaches `RewriteDefaultExpression` (#57531), which
reads the default from the bound column. branch-4.1 #66538, also in
branch-4.2, resolves these against the pinned write target too. It
rejects `DEFAULT(column)` on an Iceberg column in a statement that does
not write the table.
**The problem, and what it cost.** Reproduced on a local cluster with a
`file://` format-version 3 table: `CREATE TABLE t (id INT, c INT)`,
`ALTER TABLE t ADD COLUMN d INT DEFAULT 42`, one row `(0, 0)`, so `d =
42`.
| Statement | Master today | After this PR (same as branch-4.2) |
|---|---|---|
| `INSERT INTO t (id, c, d) VALUES (1, 1, DEFAULT(d))` | `Unknown column
'd' in 'table list'` | writes 42 |
| `INSERT INTO t (id, c, d) SELECT 2, 2, DEFAULT(d) FROM t WHERE id = 0`
| writes NULL | writes 42 |
| `INSERT INTO t (id, c, d) SELECT 3, 3, DEFAULT(d)` | `Unknown column
'd'` | writes 42 |
| `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates nothing: `d =
NULL` makes the plan an empty set | updates every row, `d = 42` is
pushed down to the scan |
| `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d =
DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | matches nothing |
updates row 0 |
| `SELECT DEFAULT(d) FROM t` | NULL | `DEFAULT(d) must reference a
column of the table that this statement writes` |
The `UPDATE` and `MERGE` cases fail silently: the statement succeeds and
changes nothing.
**How this PR fixes it.**
- fe-core cannot tell from a column whether its connector keeps defaults
in the read schema. Hive does: an HMS default shows up in both schemas.
Iceberg does not. The new connector capability
`SUPPORTS_WRITE_ONLY_DEFAULTS` says which kind a table is. The Iceberg
connector declares it. The Hive connector passes it on to iceberg-on-HMS
tables, like the other per-table capabilities, but only through the
two-argument `getTableSchema`. Since #69299, a statement that pins the
table at its latest snapshot reads the schema from the snapshot
overload, which reflects none of them, so on an iceberg-on-HMS table
such a statement keeps master's results (for example, `UPDATE t SET c =
100 WHERE d = DEFAULT(d)` still changes no row). Reflecting the
sibling's capabilities there changes planning for all of them and is
left to a follow-up PR.
- For a target that declares it, `InsertUtils.normalizePlan` resolves
`DEFAULT(column)` in `INSERT ... VALUES` and in the select lists of
`INSERT ... SELECT`, including the replacements of `SELECT *
REPLACE(...)`, against the pinned write schema. It does this by name,
before the source relations are bound, so the name always refers to a
target column, as in branch-4.2. In MySQL, `DEFAULT(column)` in a
`SELECT` refers to a source column instead. A derived table in `FROM` is
a query of its own and is left alone: `DEFAULT(column)` in it still
names a column of its own `FROM` clause.
- `RewriteDefaultExpression` takes the default for a `DEFAULT(column)`
bound to such a table from the pinned write column with the same field
id. That covers the `WHERE` of `UPDATE` and the `ON` of `MERGE`. A MERGE
resolves it whatever its WHEN branches, also when they all delete, as
branch-4.2 does. If the statement pinned no write schema for that table,
it fails. A `DELETE` pins the writer schema too, for its delete sink,
but it writes no row values, so `RowLevelDmlCommand` marks its target
and `DEFAULT(column)` on it fails with `DEFAULT(d) cannot be used in a
DELETE: the defaults of this table apply only to the rows that a
statement writes`, as in branch-4.2, whose DELETE installs no write
schema context. Master reads NULL there, so `DELETE FROM t WHERE d =
DEFAULT(d)` deletes nothing; resolved to the write-default it would
delete every row with `d = 42`.
- A resolved default is cast to the type of its writer column before the
expression around it is bound, as `RewriteDefaultExpression` already
does for a bound `DEFAULT(column)`. The parsed literal `-1` is a
TINYINT, so without the cast `bit_count(DEFAULT(c))` on an INT column
would count 8 bits instead of 32. The external `UPDATE`, `MERGE` and
changelog paths use the same helper. `RewriteDefaultExpression` rejects
a bound `DEFAULT(column)` whose writer column has another type than the
read column the statement was bound with, which happens only when the
table changed outside Doris after Doris cached it, and asks to refresh
the table: the expressions around it were bound against the read type.
- Hive, Paimon and internal tables do not declare the capability, so
they behave as before.
- The new enum constant changes the connector plugin API surface, so
`connector.plugin.api.version` goes from 26.0 to 27.0, as for the other
recent changes to that surface (#69299, #69588, #69298, #69683 and
#69835 took 22.0 to 26.0).
**Where `DEFAULT(column)` resolves on an Iceberg target (direct Iceberg
catalogs).**
| Statement | Position of `DEFAULT(column)` | Result | branch-4.2 |
|---|---|---|---|
| `INSERT` / `INSERT OVERWRITE` | `VALUES` rows | the target column's
write-default, typed as the column | same |
| | the INSERT's own select lists (with or without `FROM`, set operation
branches, `SELECT * REPLACE`) | the target column's write-default, by
name, before binding | same, except `* REPLACE`, which branch-4.2 binds
to the source |
| | `GROUP BY` / `ROLLUP` outputs, a derived table in `FROM`, `WITH`
definitions, subqueries, `WHERE` / `ON` | bound like any
`DEFAULT(column)` (last row) | same, except derived tables, which
branch-4.2 resolves against the target |
| | a `VALUES` branch of a set operation | fails with an unknown column,
as on master | same |
| | a qualifier other than the target (`DEFAULT(src.c)`,
`DEFAULT(ctl.db.t.c)`) in the positions above | fails: `column in
assignment list is invalid` | same |
| `UPDATE` | `SET` / `WHERE` | write-default | same |
| `MERGE` | `ON` and `WHEN ... AND` conditions, whatever the WHEN
branches, also when they all delete | write-default | same |
| | `UPDATE SET` / `INSERT VALUES` | write-default | same |
| `DELETE` (also `DELETE ... USING`) | anywhere | fails: `cannot be used
in a DELETE` | fails: its DELETE has no write schema context |
| any | a bound `DEFAULT(x.c)` | `x` is the written table: its
write-default by field id; an OLAP or Hive table: that column's default;
another Iceberg table: fails with `must reference a column of the table
that this statement writes` | same |
A bound `DEFAULT(column)` whose writer column has another type than the
read column fails and asks to refresh the table. Where `DEFAULT(column)`
is bound rather than resolved by name (the third row), it keeps its
meaning from before this PR and names a column of its own `FROM` clause,
as on master and branch-4.2: for example in `INSERT INTO t SELECT id,
DEFAULT(d) FROM src GROUP BY id, d` with an OLAP or Hive `src`, it takes
the default of `src.d`.
**Results.**
- Local cluster, with the table above: the table shows what each
statement did before and after. `EXPLAIN UPDATE t SET c = 5 WHERE d =
DEFAULT(d)` goes from an empty set to a scan with `d = 42` pushed down
to Iceberg.
- Unit tests: 6 new tests in `InsertUtilsTest`, 6 in the new
`RewriteDefaultExpressionTest` and 1 in `RowLevelDmlCommandTest`. They
cover VALUES, and INSERT ... SELECT both from a staging relation whose
own column is also `name` and without FROM: each resolves to the pinned
default, and an unknown column fails. `DEFAULT(column)` inside a derived
table, qualified or not and also under a UNION branch, is left to
binding. A `SELECT * REPLACE(DEFAULT(name) AS name)` replacement
resolves like the rest of the select list and keeps the star's qualifier
and `EXCEPT` list. The pinned write column wins over the read column. A
statement that writes nothing fails. A table without the capability is
unchanged. With master's `InsertUtils` and `RewriteDefaultExpression`,
the 5 tests of the fix fail and the 2 tests of the unchanged path pass.
The `SELECT * REPLACE` test fails with this PR's previous `InsertUtils`,
where the replacement stays `DEFAULT(name)`. `IcebergConnectorTest`,
`HiveConnectorMetadataSiblingDelegationTest` and
`ConnectorPluginSurfaceTest` pass. So do the fe-core tests around the
write schema: `ConnectorWriteSchemaUtilsTest`,
`ExternalRowLevelUpdatePlanBuilderTest`,
`ExternalRowLevelMergePlanBuilderTest`,
`ChangelogRowLevelDmlTransformTest` and `PluginDrivenExternalTableTest`.
Two of the tests pin the typing: `bit_count(DEFAULT(bits))` gets the
default as an INT in VALUES and in SELECT, and fails without the cast; a
BIGINT writer column over an INT read column is rejected, directly and
under an IF bound to INT. `DEFAULT(column)` on the target of a DELETE
fails even with a pinned write-default, and only a DELETE marks its
target (an UPDATE does not); skipping the check, or marking UPDATE
instead of DELETE, makes exactly these two tests fail.
`PositionDeleteRowLevelDmlTransformTest`,
`PhysicalPlanTranslatorIcebergRowLevelDmlTest`, `DeleteFromCommandTest`
and `UpdateCommandTest` pass.
- Regression: `test_iceberg_initial_defaults` now writes rows 20 to 22
with `DEFAULT(default_int)` in VALUES and in INSERT ... SELECT with and
without FROM. It then updates them with `WHERE default_int =
DEFAULT(default_int)` and `MERGE ... ON ... = DEFAULT(...)`, for Parquet
and ORC, and checks the rows in the `.out`. A plain `SELECT
DEFAULT(default_int)` must fail, and so must a `DELETE ... WHERE
default_int = DEFAULT(default_int)` on rows 20 to 22, which hold the
write-default. The suite passed locally before the DELETE block was
added, and the whole suite, DELETE block included, has since passed in
two External regression runs on CI. With master's FE it fails at the
first of these statements: `Unknown column 'default_int' in 'table
list'`.
---
.../connector/hive/HiveConnectorMetadata.java | 5 +-
...HiveConnectorMetadataSiblingDelegationTest.java | 6 +-
.../doris/connector/iceberg/IcebergConnector.java | 7 +-
.../connector/iceberg/IcebergConnectorTest.java | 12 ++
.../doris/connector/spi/ConnectorCapability.java | 17 +-
.../connector/spi/ConnectorPluginSurfaceTest.java | 5 +-
.../test/resources/connector-plugin-surface.txt | 1 +
fe/fe-connector/pom.xml | 2 +-
.../plugin/PluginDrivenExternalTable.java | 11 ++
.../org/apache/doris/nereids/StatementContext.java | 12 ++
.../expression/rules/RewriteDefaultExpression.java | 37 ++++
.../plans/commands/ConnectorWriteSchemaUtils.java | 6 +-
.../trees/plans/commands/RowLevelDmlCommand.java | 15 +-
.../trees/plans/commands/insert/InsertUtils.java | 92 ++++++++-
.../rules/RewriteDefaultExpressionTest.java | 152 +++++++++++++++
.../commands/ConnectorWriteSchemaUtilsTest.java | 11 +-
.../plans/commands/RowLevelDmlCommandTest.java | 21 ++
.../plans/commands/insert/InsertUtilsTest.java | 212 +++++++++++++++++++++
.../iceberg/test_iceberg_initial_defaults.out | 10 +
.../iceberg/test_iceberg_initial_defaults.groovy | 39 ++++
20 files changed, 648 insertions(+), 25 deletions(-)
diff --git
a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
index a588b804927..3fefe4fddde 100644
---
a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
+++
b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java
@@ -623,7 +623,8 @@ public class HiveConnectorMetadata implements
ConnectorMetadata {
ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE,
ConnectorCapability.SUPPORTS_FIELD_ID_ACCESS_PATH,
ConnectorCapability.SUPPORTS_STORAGE_PREDICATE_PRUNING,
- ConnectorCapability.SUPPORTS_NESTED_COLUMN_SCHEMA_CHANGE));
+ ConnectorCapability.SUPPORTS_NESTED_COLUMN_SCHEMA_CHANGE,
+ ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS));
/**
* Reflects the {@link #SIBLING_INHERITABLE_CAPABILITIES} subset of the
owning sibling connector's
@@ -633,7 +634,7 @@ public class HiveConnectorMetadata implements
ConnectorMetadata {
* (hive) connector-wide set OR the table's own set, and NEVER consults
the sibling connector directly, so
* without this reflection an iceberg-on-HMS table would silently lose
every such capability the iceberg
* sibling declares connector-wide (auto-analyze / Top-N lazy /
nested-column prune / storage predicate
- * pruning / nested column DDL).
+ * pruning / nested column DDL / write-only defaults).
* Returns the sibling schema unchanged when nothing is inherited and the
sibling declared nothing itself
* (e.g. a hudi sibling, which declares no capabilities at all).
*/
diff --git
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
index d2f340f0f31..77a7b1529eb 100644
---
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
+++
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java
@@ -525,7 +525,8 @@ public class HiveConnectorMetadataSiblingDelegationTest {
ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE,
ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE,
ConnectorCapability.SUPPORTS_FIELD_ID_ACCESS_PATH,
- ConnectorCapability.SUPPORTS_STORAGE_PREDICATE_PRUNING);
+ ConnectorCapability.SUPPORTS_STORAGE_PREDICATE_PRUNING,
+ ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS);
HiveConnectorMetadata md = new HiveConnectorMetadata(null,
HiveTestProperties.minimal(), new FakeConnectorContext(),
SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED,
handle -> new SiblingOwner(new
CapabilityDeclaringSiblingConnector(siblingCaps),
@@ -533,6 +534,9 @@ public class HiveConnectorMetadataSiblingDelegationTest {
ConnectorTableSchema schema = md.getTableSchema(session,
foreignHandle);
Set<ConnectorCapability> reflected = schema.getTableCapabilities();
+
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS),
+ "an iceberg-on-HMS table must inherit write-only defaults, or
DEFAULT(column) on it answers "
+ + "NULL from the read schema");
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE),
"auto-analyze must survive the delegation as a per-table
capability");
Assertions.assertTrue(reflected.contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE),
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
index 5d3c3ef3095..0d21e2a27bc 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java
@@ -897,6 +897,10 @@ public class IcebergConnector implements Connector {
// (CREATE TABLE ... ORDER BY (...)). fe-core admits the ORDER BY
clause for a plugin-driven CREATE TABLE
// ONLY under this capability (replacing the legacy engine-name
"iceberg" gate); the sort-column validation
// (existence / type / duplicates) runs connector-side in
IcebergConnectorMetadata.createTable.
+ // SUPPORTS_WRITE_ONLY_DEFAULTS: an iceberg write-default applies to
new rows only, so it lives in the
+ // writer schema (IcebergWriteSchemaContext) and not in the read
schema. fe-core resolves DEFAULT(column)
+ // against the writer schema pinned for the write target and rejects
it in a plain SELECT, instead of
+ // answering NULL from the read schema.
EnumSet<ConnectorCapability> capabilities =
EnumSet.of(ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT,
ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE,
ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE,
@@ -907,7 +911,8 @@ public class IcebergConnector implements Connector {
ConnectorCapability.SUPPORTS_STORAGE_PREDICATE_PRUNING,
ConnectorCapability.SUPPORTS_METADATA_PRELOAD,
ConnectorCapability.SUPPORTS_SORT_ORDER,
- ConnectorCapability.SUPPORTS_NESTED_COLUMN_SCHEMA_CHANGE);
+ ConnectorCapability.SUPPORTS_NESTED_COLUMN_SCHEMA_CHANGE,
+ ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS);
// SUPPORTS_USER_SESSION: only a REST catalog configured
iceberg.rest.session=user projects the querying
// user's delegated credential onto a per-request Iceberg REST
SessionCatalog (#63068 re-migration). This
// gates FE credential injection + shared-cache bypass; every other
flavor/config authenticates with a
diff --git
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java
index ee671040606..09c99edc77a 100644
---
a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java
+++
b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java
@@ -200,6 +200,18 @@ public class IcebergConnectorTest {
+ "materialization");
}
+ @Test
+ public void declaresWriteOnlyDefaultsCapability() {
+ // WHY: an iceberg write-default is carried by the writer schema only,
so the read schema's columns have no
+ // default. fe-core resolves DEFAULT(column) against the pinned writer
schema (and rejects it in a plain
+ // SELECT) ONLY under this capability. MUTATION: dropping the
capability -> INSERT ... SELECT DEFAULT(c)
+ // writes NULL and UPDATE ... WHERE c = DEFAULT(c) silently matches
nothing -> red.
+ IcebergConnector connector = new
IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext());
+ Assertions.assertTrue(connector.getCapabilities()
+
.contains(ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS),
+ "iceberg must declare SUPPORTS_WRITE_ONLY_DEFAULTS so
DEFAULT(column) resolves to the write default");
+ }
+
@Test
public void declaresShowCreateDdlCapability() {
// WHY: legacy IcebergExternalTable rendered LOCATION + PROPERTIES +
PARTITION BY + ORDER BY in SHOW
diff --git
a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorCapability.java
b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorCapability.java
index 97e8d546154..2c626b5fd1d 100644
---
a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorCapability.java
+++
b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorCapability.java
@@ -304,5 +304,20 @@ public enum ConnectorCapability {
* data tables while declining it on the subset of system tables whose
readers cannot observe a
* selected snapshot.</p>
*/
- SUPPORTS_SCAN_PARAM_OPTIONS
+ SUPPORTS_SCAN_PARAM_OPTIONS,
+ /**
+ * Indicates the connector's column defaults apply to writes only: they
are carried by the
+ * request-scoped writer schema ({@code
ConnectorWritePlanProvider.getWriteColumns}), not by the read
+ * schema (e.g. Iceberg {@code write-default}).
+ *
+ * <p>fe-core then resolves {@code DEFAULT(column)} against the writer
schema a statement pinned for its
+ * write target: in INSERT VALUES, by name against the target in INSERT
... SELECT, and in the
+ * predicates of UPDATE / MERGE on the target. Anywhere else, e.g. in a
plain SELECT, {@code
+ * DEFAULT(column)} on such a table is rejected instead of answering NULL
from the read schema.
+ * Connectors whose read schema carries the column defaults (hive) must
NOT declare it.</p>
+ *
+ * <p><b>Scope: catalog-wide OR per-table.</b> An iceberg-on-HMS table
inherits it through the per-table
+ * set the gateway reflects from its sibling, exactly like {@link
#SUPPORTS_NESTED_COLUMN_PRUNE}.</p>
+ */
+ SUPPORTS_WRITE_ONLY_DEFAULTS
}
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
index 6ea3635b1a2..4f1218df17f 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
+++
b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
@@ -88,8 +88,9 @@ public class ConnectorPluginSurfaceTest {
Assertions.assertNotNull(in, "missing connector plugin API version
resource");
version.load(in);
}
- // Major 26 adds scan cancellation notifications and task removal to
the scheduler surface.
- Assertions.assertEquals("26.0", version.getProperty("api.version"));
+ // Major 26 adds scan cancellation notifications and task removal to
the scheduler surface;
+ // major 27 adds the ConnectorCapability#SUPPORTS_WRITE_ONLY_DEFAULTS
constant.
+ Assertions.assertEquals("27.0", version.getProperty("api.version"));
}
/** Root entry points plus provider/handle types returned to connector
plugins. */
diff --git
a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
index 725d612521c..35ad6b851f1 100644
---
a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
+++
b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
@@ -35,6 +35,7 @@
org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_SYS_TABLE_NESTE
org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_TOPN_LAZY_MATERIALIZE
org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_USER_SESSION
org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_VIEW
+org.apache.doris.connector.spi.ConnectorCapability#enum:SUPPORTS_WRITE_ONLY_DEFAULTS
org.apache.doris.connector.spi.ConnectorColumn#equals(java.lang.Object):boolean
org.apache.doris.connector.spi.ConnectorColumn#getComment():java.lang.String
org.apache.doris.connector.spi.ConnectorColumn#getDefaultValue():java.lang.String
diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml
index cbc125e18ac..f5394110e54 100644
--- a/fe/fe-connector/pom.xml
+++ b/fe/fe-connector/pom.xml
@@ -55,7 +55,7 @@ under the License.
of the latter two means bumping this property as well (and
fe-extension-spi means bumping
all five families).
-->
- <connector.plugin.api.version>26.0</connector.plugin.api.version>
+ <connector.plugin.api.version>27.0</connector.plugin.api.version>
</properties>
<modules>
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java
index 6e3115668d0..47b1d6fc8cf 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalTable.java
@@ -475,6 +475,17 @@ public class PluginDrivenExternalTable extends
ExternalTable {
return hasCapability(ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE);
}
+ /**
+ * Returns whether THIS table's column defaults apply to writes only, i.e.
only the request-scoped writer
+ * schema ({@link #resolveWriteColumns}) carries them. {@code InsertUtils}
and {@code RewriteDefaultExpression}
+ * then resolve {@code DEFAULT(column)} against the writer schema a write
statement pinned, and reject it
+ * anywhere else instead of answering NULL from the read schema. Resolved
per-table via {@link #hasCapability}
+ * so an iceberg-on-HMS table inherits it, mirroring {@link
#supportsNestedColumnSchemaChange}.
+ */
+ public boolean hasWriteOnlyDefaults() {
+ return hasCapability(ConnectorCapability.SUPPORTS_WRITE_ONLY_DEFAULTS);
+ }
+
/**
* Whether this table supports a table-scoped capability, resolved
connector-wide OR per-table. A
* uniform-format connector (iceberg — every table orc/parquet) declares
the capability for all its tables
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
index 2a74e0b7cb1..bf30f1d4949 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java
@@ -293,6 +293,9 @@ public class StatementContext implements Closeable {
// The identity is captured by the same provider call chain as
connectorWriteSchemas. Keeping it alongside
// the columns prevents bind-time expressions and the later sink fence
from naming different generations.
private final Map<Long, String> connectorWriteMetadataIdentities = new
HashMap<>();
+ // Connector targets of a DELETE. Their writer schema is pinned for the
delete sink, but the statement writes no
+ // row values, so a write-only default in that schema does not apply to
DEFAULT(column) on them.
+ private final Set<Long> connectorDeleteTargets = new HashSet<>();
// for create view support in nereids
// key is the start and end position of the sql substring that needs to be
@@ -591,6 +594,14 @@ public class StatementContext implements Closeable {
return Optional.ofNullable(connectorWriteSchemas.get(tableId));
}
+ public void addConnectorDeleteTarget(long tableId) {
+ connectorDeleteTargets.add(tableId);
+ }
+
+ public boolean isConnectorDeleteTarget(long tableId) {
+ return connectorDeleteTargets.contains(tableId);
+ }
+
public void setConnectorWriteMetadataIdentity(long tableId, String
identity) {
connectorWriteMetadataIdentities.put(tableId, identity);
}
@@ -914,6 +925,7 @@ public class StatementContext implements Closeable {
this.connectorStatementScope = null;
this.connectorWriteSchemas.clear();
this.connectorWriteMetadataIdentities.clear();
+ this.connectorDeleteTargets.clear();
changelogReferenceTsoRequired = false;
changelogReferenceTso = null;
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpression.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpression.java
index 6dc8ab7c72c..d0a3a455097 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpression.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpression.java
@@ -18,7 +18,10 @@
package org.apache.doris.nereids.rules.expression.rules;
import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.datasource.plugin.PluginDrivenExternalTable;
import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.analyzer.UnboundFunction;
import org.apache.doris.nereids.exceptions.AnalysisException;
import org.apache.doris.nereids.parser.NereidsParser;
@@ -46,6 +49,9 @@ import java.util.Optional;
* 1. DEFAULT(column) with default value -> the actual default value expression
* 2. DEFAULT(column) with no default value but nullable -> NULL
* 3. DEFAULT(column) with no default value and not nullable -> error
+ * A table whose defaults apply to writes only takes them from the writer
schema that the statement
+ * pinned for its write target; DEFAULT(column) on any other such table, or on
the target of a DELETE,
+ * which writes no row values, is an error.
*/
public class RewriteDefaultExpression implements ExpressionPatternRuleFactory {
@@ -82,6 +88,29 @@ public class RewriteDefaultExpression implements
ExpressionPatternRuleFactory {
+ column.getName() + "'");
}
+ Optional<TableIf> originalTable = slotRef.getOriginalTable();
+ if (originalTable.filter(PluginDrivenExternalTable.class::isInstance)
+ .filter(table -> ((PluginDrivenExternalTable)
table).hasWriteOnlyDefaults())
+ .isPresent()) {
+ StatementContext statementContext =
context.cascadesContext.getStatementContext();
+ if
(statementContext.isConnectorDeleteTarget(originalTable.get().getId())) {
+ throw new AnalysisException("DEFAULT(" + slotRef.toSql() + ")
cannot be used in a DELETE: the"
+ + " defaults of this table apply only to the rows that
a statement writes");
+ }
+ column = pinnedWriteColumn(statementContext, originalTable.get(),
column)
+ .orElseThrow(() -> new AnalysisException("DEFAULT(" +
slotRef.toSql()
+ + ") must reference a column of the table that
this statement writes"));
+ // The expressions around DEFAULT(column) were bound against the
read column, so a default of another type
+ // cannot stand in for it. The types differ when the table changed
outside Doris after its read schema was
+ // cached, e.g. by a type promotion.
+ DataType writeType = DataType.fromCatalogType(column.getType());
+ if (!writeType.equals(targetType)) {
+ throw new AnalysisException("DEFAULT(" + slotRef.toSql() + ")
reads the column as "
+ + targetType.toSql() + " but the table now writes it
as " + writeType.toSql()
+ + "; the table changed outside Doris, refresh it and
retry");
+ }
+ }
+
String defaultValueSql = column.getDefaultValueSql();
if (defaultValueSql == null) {
if (column.isAllowNull()) {
@@ -101,4 +130,12 @@ public class RewriteDefaultExpression implements
ExpressionPatternRuleFactory {
return TypeCoercionUtils.castIfNotSameType(defaultValueExpr,
targetType);
}
+
+ private static Optional<Column> pinnedWriteColumn(StatementContext
statementContext, TableIf table,
+ Column readColumn) {
+ return statementContext.getConnectorWriteSchema(table.getId())
+ .flatMap(columns -> columns.stream()
+ .filter(column -> column.getUniqueId() ==
readColumn.getUniqueId())
+ .findFirst());
+ }
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtils.java
index c4a8d94f9b2..a7f063eaba6 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtils.java
@@ -28,6 +28,8 @@ import org.apache.doris.nereids.trees.expressions.Default;
import org.apache.doris.nereids.trees.expressions.DefaultValueSlot;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.types.DataType;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
@@ -101,7 +103,9 @@ public final class ConnectorWriteSchemaUtils {
throw new AnalysisException("Cannot find column information
for DEFAULT("
+ reference.toSql() + ")");
}
- return resolveDefault(column);
+ // The parsed default takes the literal's own type (-1 is a
TINYINT); the expression around DEFAULT(column)
+ // must bind against the column's type, as the bound DEFAULT
rewrite does.
+ return TypeCoercionUtils.castUnbound(resolveDefault(column),
DataType.fromCatalogType(column.getType()));
});
}
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommand.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommand.java
index cb1884eb572..5917710bc9e 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommand.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommand.java
@@ -82,7 +82,7 @@ public class RowLevelDmlCommand {
public void run(ConnectContext ctx, StmtExecutor stmtExecutor) throws
Exception {
TableIf table = args.getTable();
transform.checkMode(table, op);
- ConnectorWriteSchemaUtils.pinAndGet(ctx, (ExternalTable) table);
+ pinWriteSchema(ctx, table);
long previousTargetTableId = ctx.getSyntheticWriteColTargetTableId();
ctx.setSyntheticWriteColTargetTableId(table.getId());
try {
@@ -154,7 +154,7 @@ public class RowLevelDmlCommand {
public Plan getExplainPlan(ConnectContext ctx) {
TableIf table = args.getTable();
transform.checkMode(table, op);
- ConnectorWriteSchemaUtils.pinAndGet(ctx, (ExternalTable) table);
+ pinWriteSchema(ctx, table);
long previousTargetTableId = ctx.getSyntheticWriteColTargetTableId();
ctx.setSyntheticWriteColTargetTableId(table.getId());
try {
@@ -164,6 +164,17 @@ public class RowLevelDmlCommand {
}
}
+ /**
+ * Pin the target's writer schema for the statement. A DELETE needs it for
its delete sink but writes no row
+ * values, so its target is also marked: DEFAULT(column) there gets no
write-only default from the schema.
+ */
+ private void pinWriteSchema(ConnectContext ctx, TableIf table) {
+ ConnectorWriteSchemaUtils.pinAndGet(ctx, (ExternalTable) table);
+ if (op == RowLevelDmlOp.DELETE) {
+ ctx.getStatementContext().addConnectorDeleteTarget(table.getId());
+ }
+ }
+
/**
* The begin→finalize window, guarded like {@code
InsertIntoTableCommand}'s prepare step:
* {@code beginTransaction} registers the transaction with the connector
transaction manager and the
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
index a4f1d1600ed..7f5803b0ef7 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java
@@ -29,6 +29,7 @@ import org.apache.doris.catalog.Table;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.Config;
import org.apache.doris.common.util.DebugPointUtil;
+import org.apache.doris.datasource.ExternalTable;
import
org.apache.doris.datasource.connector.converter.ConnectorWriteValueConverter;
import org.apache.doris.datasource.plugin.PluginDrivenExternalTable;
import org.apache.doris.foundation.format.FormatOptions;
@@ -41,6 +42,7 @@ import
org.apache.doris.nereids.analyzer.UnboundConnectorTableSink;
import org.apache.doris.nereids.analyzer.UnboundDictionarySink;
import org.apache.doris.nereids.analyzer.UnboundFunction;
import org.apache.doris.nereids.analyzer.UnboundInlineTable;
+import org.apache.doris.nereids.analyzer.UnboundOneRowRelation;
import org.apache.doris.nereids.analyzer.UnboundSlot;
import org.apache.doris.nereids.analyzer.UnboundStar;
import org.apache.doris.nereids.analyzer.UnboundTableSink;
@@ -62,9 +64,12 @@ import
org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.algebra.InlineTable;
+import org.apache.doris.nereids.trees.plans.commands.ConnectorWriteSchemaUtils;
import org.apache.doris.nereids.trees.plans.commands.info.DMLCommandType;
import org.apache.doris.nereids.trees.plans.logical.LogicalInlineTable;
import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSubQueryAlias;
import org.apache.doris.nereids.trees.plans.logical.UnboundLogicalSink;
import org.apache.doris.nereids.types.AggStateType;
import org.apache.doris.nereids.types.DataType;
@@ -370,7 +375,12 @@ public class InsertUtils {
}
Plan query = unboundLogicalSink.child();
checkGeneratedColumnForInsertIntoSelect(table, unboundLogicalSink,
insertCtx);
+ Optional<List<Column>> writeOnlyDefaultColumns =
writeOnlyDefaultColumns(table);
if (!(query instanceof UnboundInlineTable)) {
+ if (writeOnlyDefaultColumns.isPresent()) {
+ return plan.withChildren(resolveSelectDefaultReferences(
+ query, writeOnlyDefaultColumns, table,
unboundLogicalSink.getNameParts()));
+ }
return plan;
}
@@ -446,7 +456,8 @@ public class InsertUtils {
null, rewriteContext, strictCast);
} else {
DataType targetType =
DataType.fromCatalogType(sameNameColumn.getType());
- NamedExpression value = values.get(i);
+ NamedExpression value =
resolveDefaultReferences(values.get(i),
+ writeOnlyDefaultColumns, table,
unboundLogicalSink.getNameParts());
if (sameNameColumn.isTtlColumn() &&
targetType.isBigIntType()) {
value = new UnboundAlias(new RowTtlExpiration(
value instanceof Alias || value
instanceof UnboundAlias
@@ -473,7 +484,9 @@ public class InsertUtils {
null, rewriteContext, strictCast);
} else {
DataType targetType =
DataType.fromCatalogType(columns.get(i).getType());
- addColumnValue(analyzer, optimizedRowConstructor,
values.get(i), targetType,
+ NamedExpression value =
resolveDefaultReferences(values.get(i),
+ writeOnlyDefaultColumns, table,
unboundLogicalSink.getNameParts());
+ addColumnValue(analyzer, optimizedRowConstructor,
value, targetType,
rewriteContext, strictCast,
columns.get(i));
}
}
@@ -485,15 +498,76 @@ public class InsertUtils {
}
static List<Column> connectorWriteSchema(TableIf table, boolean full) {
+ return pinnedConnectorWriteSchema(table).orElseGet(() ->
table.getBaseSchema(full));
+ }
+
+ private static Optional<List<Column>> pinnedConnectorWriteSchema(TableIf
table) {
ConnectContext context = ConnectContext.get();
- if (context != null && context.getStatementContext() != null) {
- Optional<List<Column>> pinned =
-
context.getStatementContext().getConnectorWriteSchema(table.getId());
- if (pinned.isPresent()) {
- return pinned.get();
- }
+ if (context == null || context.getStatementContext() == null) {
+ return Optional.empty();
+ }
+ return
context.getStatementContext().getConnectorWriteSchema(table.getId());
+ }
+
+ /**
+ * The pinned writer schema of a target whose column defaults apply to
writes only. The read schema of such a
+ * table carries no defaults, so DEFAULT(column) in an INSERT into it must
resolve against this schema.
+ */
+ private static Optional<List<Column>> writeOnlyDefaultColumns(TableIf
table) {
+ if (!(table instanceof PluginDrivenExternalTable)
+ || !((PluginDrivenExternalTable)
table).hasWriteOnlyDefaults()) {
+ return Optional.empty();
}
- return table.getBaseSchema(full);
+ return pinnedConnectorWriteSchema(table);
+ }
+
+ /** Resolve DEFAULT(column) in one inserted value by name against the
target's pinned writer schema. */
+ private static NamedExpression resolveDefaultReferences(NamedExpression
value,
+ Optional<List<Column>> writeOnlyDefaultColumns, TableIf table,
List<String> targetNameParts) {
+ if (!writeOnlyDefaultColumns.isPresent()) {
+ return value;
+ }
+ if (value instanceof UnboundStar) {
+ // SELECT * REPLACE(expr AS c) keeps its replacements in the star,
not as expression children, so
+ // resolving the star's expression tree would leave a DEFAULT(c)
there to bind against the source.
+ UnboundStar star = (UnboundStar) value;
+ return new UnboundStar(star.getQualifier(),
star.getExceptedSlots(), star.getReplacedAlias().stream()
+ .map(replace -> resolveDefaultReferences(replace,
writeOnlyDefaultColumns, table, targetNameParts))
+ .collect(ImmutableList.toImmutableList()),
star.getIndexInSqlString());
+ }
+ return (NamedExpression)
ConnectorWriteSchemaUtils.resolveDefaultReferences(value,
+ writeOnlyDefaultColumns.get(), (ExternalTable) table,
ConnectContext.get(), targetNameParts, null);
+ }
+
+ /**
+ * Resolve DEFAULT(column) in the select lists of an INSERT ... SELECT,
including the replacements of
+ * SELECT * REPLACE, by name against the target's pinned writer schema,
before the source relations are
+ * bound, so it names a target column whatever the source is.
+ * A derived table in FROM is a query of its own: DEFAULT(column) in it
names a column of its own FROM clause,
+ * as without this resolution, so it is left alone.
+ */
+ private static Plan resolveSelectDefaultReferences(Plan query,
Optional<List<Column>> writeOnlyDefaultColumns,
+ TableIf table, List<String> targetNameParts) {
+ if (query instanceof LogicalSubQueryAlias) {
+ return query;
+ }
+ Plan plan = query.withChildren(child -> resolveSelectDefaultReferences(
+ child, writeOnlyDefaultColumns, table, targetNameParts));
+ if (plan instanceof LogicalProject) {
+ LogicalProject<?> project = (LogicalProject<?>) plan;
+ return project.withProjects(project.getProjects().stream()
+ .map(projection -> resolveDefaultReferences(
+ projection, writeOnlyDefaultColumns, table,
targetNameParts))
+ .collect(ImmutableList.toImmutableList()));
+ }
+ if (plan instanceof UnboundOneRowRelation) {
+ UnboundOneRowRelation relation = (UnboundOneRowRelation) plan;
+ return new UnboundOneRowRelation(relation.getRelationId(),
relation.getProjects().stream()
+ .map(projection -> resolveDefaultReferences(
+ projection, writeOnlyDefaultColumns, table,
targetNameParts))
+ .collect(ImmutableList.toImmutableList()));
+ }
+ return plan;
}
static void pinConnectorWriteSchema(StatementContext statementContext,
TableIf targetTableIf,
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpressionTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpressionTest.java
new file mode 100644
index 00000000000..11a7705a393
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/expression/rules/RewriteDefaultExpressionTest.java
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.datasource.plugin.PluginDrivenExternalTable;
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteTestHelper;
+import org.apache.doris.nereids.rules.expression.ExpressionRuleExecutor;
+import org.apache.doris.nereids.trees.expressions.Default;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.If;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.collect.ImmutableList;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+public class RewriteDefaultExpressionTest extends ExpressionRewriteTestHelper {
+
+ private static final long TABLE_ID = 17L;
+ private static final int FIELD_ID = 3;
+
+ @BeforeEach
+ public void setUp() {
+ executor = new
ExpressionRuleExecutor(ImmutableList.of(bottomUp(RewriteDefaultExpression.INSTANCE)));
+ }
+
+ @Test
+ public void writeOnlyDefaultResolvesAgainstPinnedWriterSchema() {
+ // UPDATE t ... WHERE score = DEFAULT(score): the scanned column
carries no write-only default, the writer
+ // schema pinned for the statement's target does.
+ SlotReference score = scoreSlot(true);
+ pinWriterColumn("9");
+
+ Expression rewritten = executor.rewrite(new Default(score), context);
+
+ Assertions.assertEquals(
+
TypeCoercionUtils.castIfNotSameType(PARSER.parseExpression("9"),
IntegerType.INSTANCE), rewritten);
+ }
+
+ @Test
+ public void writeOnlyDefaultOfAWriterColumnOfAnotherTypeIsRejected() {
+ // The read schema can still hold the INT column after it was promoted
to BIGINT outside Doris, while the
+ // writer schema pinned for the statement has the BIGINT column and
its new default, which an INT cannot hold.
+ SlotReference score = scoreSlot(true);
+ pinWriterColumn(PrimitiveType.BIGINT, "4900000001");
+
+ AnalysisException e = Assertions.assertThrows(AnalysisException.class,
+ () -> executor.rewrite(new Default(score), context));
+
+ Assertions.assertTrue(e.getMessage().contains("DEFAULT(score) reads
the column as INT but the table now writes"
+ + " it as BIGINT; the table changed outside Doris, refresh it
and retry"), e.getMessage());
+ }
+
+ @Test
+ public void
writeOnlyDefaultOfAnotherTypeIsRejectedInsideAnExpressionBoundToTheReadType() {
+ // UPDATE t ... WHERE IF(TRUE, DEFAULT(score), 0) = 4900000001: IF was
bound with the INT of the read column,
+ // so substituting the BIGINT default would later fold into
CAST(4900000001 AS INT), NULL in non-strict mode.
+ SlotReference score = scoreSlot(true);
+ pinWriterColumn(PrimitiveType.BIGINT, "4900000001");
+ If ifExpr = new If(BooleanLiteral.TRUE, new Default(score), new
IntegerLiteral(0));
+ Assertions.assertEquals(IntegerType.INSTANCE, ifExpr.getDataType());
+
+ AnalysisException e = Assertions.assertThrows(AnalysisException.class,
+ () -> executor.rewrite(ifExpr, context));
+
+ Assertions.assertTrue(e.getMessage().contains("refresh it and retry"),
e.getMessage());
+ }
+
+ @Test
+ public void writeOnlyDefaultOutsideTheWriteTargetIsRejected() {
+ // SELECT DEFAULT(score) FROM t writes nothing, so nothing carries the
write-only default.
+ SlotReference score = scoreSlot(true);
+
+ AnalysisException e = Assertions.assertThrows(AnalysisException.class,
+ () -> executor.rewrite(new Default(score), context));
+
+ Assertions.assertTrue(e.getMessage().contains(
+ "DEFAULT(score) must reference a column of the table that this
statement writes"), e.getMessage());
+ }
+
+ @Test
+ public void writeOnlyDefaultInADeleteIsRejected() {
+ // DELETE FROM t WHERE score = DEFAULT(score): the statement pins the
writer schema for its delete sink, but
+ // it writes no row values, so the write-only default 9 must not match
the rows it deletes.
+ SlotReference score = scoreSlot(true);
+ pinWriterColumn("9");
+
cascadesContext.getStatementContext().addConnectorDeleteTarget(TABLE_ID);
+
+ AnalysisException e = Assertions.assertThrows(AnalysisException.class,
+ () -> executor.rewrite(new Default(score), context));
+
+ Assertions.assertTrue(e.getMessage().contains("DEFAULT(score) cannot
be used in a DELETE: the defaults of"
+ + " this table apply only to the rows that a statement
writes"), e.getMessage());
+ }
+
+ @Test
+ public void readSchemaDefaultIsKeptWithoutWriteOnlyDefaults() {
+ // A connector whose read schema carries its defaults (hive) resolves
from the scanned column as before.
+ SlotReference score = scoreSlot(false);
+ pinWriterColumn("9");
+
+ Expression rewritten = executor.rewrite(new Default(score), context);
+
+ Assertions.assertEquals(new NullLiteral(IntegerType.INSTANCE),
rewritten);
+ }
+
+ private SlotReference scoreSlot(boolean writeOnlyDefaults) {
+ PluginDrivenExternalTable table =
Mockito.mock(PluginDrivenExternalTable.class);
+ Mockito.when(table.getId()).thenReturn(TABLE_ID);
+
Mockito.when(table.hasWriteOnlyDefaults()).thenReturn(writeOnlyDefaults);
+ Column readColumn = new Column("score", PrimitiveType.INT, true);
+ readColumn.setUniqueId(FIELD_ID);
+ return SlotReference.fromColumn(new ExprId(1), table, readColumn,
ImmutableList.of());
+ }
+
+ private void pinWriterColumn(String defaultSql) {
+ pinWriterColumn(PrimitiveType.INT, defaultSql);
+ }
+
+ private void pinWriterColumn(PrimitiveType type, String defaultSql) {
+ Column writeColumn = new Column("score", type, true);
+ writeColumn.setUniqueId(FIELD_ID);
+ writeColumn.setConnectorDefaultValueSql(defaultSql);
+
cascadesContext.getStatementContext().setConnectorWriteSchema(TABLE_ID,
ImmutableList.of(writeColumn));
+ }
+}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtilsTest.java
index b61a51c1ead..32ce0f8827c 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtilsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ConnectorWriteSchemaUtilsTest.java
@@ -36,8 +36,8 @@ import
org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.expressions.literal.DecimalV3Literal;
import org.apache.doris.nereids.trees.expressions.literal.DoubleLiteral;
import org.apache.doris.nereids.trees.expressions.literal.FloatLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
-import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.nereids.types.DecimalV3Type;
import org.apache.doris.nereids.util.TypeCoercionUtils;
@@ -96,12 +96,13 @@ public class ConnectorWriteSchemaUtilsTest extends
ParserTestBase {
@Test
public void defaultReferencesResolveAgainstPinnedWriteColumns() {
- // DEFAULT(column) in DML resolves the statement-pinned write default;
a name absent from the pinned writer
- // schema is an error rather than a guess.
+ // DEFAULT(column) in DML resolves the statement-pinned write default,
typed as its INT column rather than
+ // as the TINYINT the literal parses to; a name absent from the pinned
writer schema is an error rather than
+ // a guess.
ExternalTable target = targetTable();
List<Column> pinned = ImmutableList.of(pinnedColumn("score", 1, "9"));
- Assertions.assertEquals(new TinyIntLiteral((byte) 9),
ConnectorWriteSchemaUtils.resolveDefaultReferences(
+ Assertions.assertEquals(new IntegerLiteral(9),
ConnectorWriteSchemaUtils.resolveDefaultReferences(
new Default(new UnboundSlot("score")), pinned, target,
ConnectContext.get(),
TARGET_NAME_PARTS, null));
AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
@@ -121,7 +122,7 @@ public class ConnectorWriteSchemaUtilsTest extends
ParserTestBase {
Column replacement = new Column("reused_name",
ScalarType.createType(PrimitiveType.INT));
replacement.setUniqueId(8);
- Assertions.assertEquals(new TinyIntLiteral((byte) 9),
ConnectorWriteSchemaUtils.resolveDefaultReferences(
+ Assertions.assertEquals(new IntegerLiteral(9),
ConnectorWriteSchemaUtils.resolveDefaultReferences(
new Default(boundSlot(target, pinnedField)), pinned, target,
ConnectContext.get(),
TARGET_NAME_PARTS, null));
AnalysisException exception =
Assertions.assertThrows(AnalysisException.class,
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommandTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommandTest.java
index 3d168db1ce2..d323e41e66e 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommandTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlCommandTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.nereids.trees.plans.commands;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.Status;
import org.apache.doris.common.UserException;
+import org.apache.doris.datasource.ExternalTable;
import org.apache.doris.nereids.CascadesContext;
import org.apache.doris.nereids.NereidsPlanner;
import org.apache.doris.nereids.PlannerHook;
@@ -34,6 +35,7 @@ import org.apache.doris.qe.Coordinator;
import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.thrift.TStatusCode;
+import com.google.common.collect.ImmutableList;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.ArgumentCaptor;
@@ -137,6 +139,25 @@ public class RowLevelDmlCommandTest {
Mockito.verify(planner,
Mockito.never()).setAnalyzedPlan(Mockito.any());
}
+ @Test
+ public void onlyADeleteMarksItsTargetAsWritingNoRowValues() {
+ // Every row-level DML pins the target's writer schema, a DELETE for
its delete sink. Only the DELETE writes
+ // no row values, so only its target is marked and keeps
DEFAULT(column) off the write-only defaults.
+ ExternalTable target = Mockito.mock(ExternalTable.class);
+ Mockito.when(target.getId()).thenReturn(7L);
+ StatementContext statementContext = new StatementContext();
+ ConnectContext ctx = Mockito.mock(ConnectContext.class);
+ Mockito.when(ctx.getStatementContext()).thenReturn(statementContext);
+
+ new RowLevelDmlCommand(transform, RowLevelDmlArgs.forUpdate(target,
ImmutableList.of("t"), null,
+ ImmutableList.of(), null),
RowLevelDmlOp.UPDATE).getExplainPlan(ctx);
+ Assertions.assertFalse(statementContext.isConnectorDeleteTarget(7L));
+
+ new RowLevelDmlCommand(transform, RowLevelDmlArgs.forDelete(target,
ImmutableList.of("t"), null, false,
+ ImmutableList.of(), null),
RowLevelDmlOp.DELETE).getExplainPlan(ctx);
+ Assertions.assertTrue(statementContext.isConnectorDeleteTarget(7L));
+ }
+
@Test
public void retainedCancellationIsReplayedBeforeTheMetadataDelete() {
// The row-level path replays the retained cancellation when the
coordinator is published. The metadata
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtilsTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtilsTest.java
index e624fecc4b0..1c88a19d093 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtilsTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtilsTest.java
@@ -28,17 +28,36 @@ import org.apache.doris.nereids.StatementContext;
import org.apache.doris.nereids.analyzer.UnboundAlias;
import org.apache.doris.nereids.analyzer.UnboundConnectorTableSink;
import org.apache.doris.nereids.analyzer.UnboundFunction;
+import org.apache.doris.nereids.analyzer.UnboundInlineTable;
+import org.apache.doris.nereids.analyzer.UnboundOneRowRelation;
+import org.apache.doris.nereids.analyzer.UnboundSlot;
+import org.apache.doris.nereids.analyzer.UnboundStar;
import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.parser.NereidsParser;
+import org.apache.doris.nereids.trees.expressions.Default;
+import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.expressions.literal.TinyIntLiteral;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.RelationId;
+import org.apache.doris.nereids.trees.plans.logical.LogicalInlineTable;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSubQueryAlias;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
import org.apache.doris.nereids.trees.plans.logical.UnboundLogicalSink;
+import org.apache.doris.nereids.types.IntegerType;
+import org.apache.doris.nereids.util.MemoTestUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+import org.apache.doris.qe.ConnectContext;
import com.google.common.collect.ImmutableList;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
+import java.util.List;
import java.util.Optional;
/**
@@ -47,6 +66,7 @@ import java.util.Optional;
public class InsertUtilsTest {
private static final int MAX_TOTAL_BYTES = 512;
+ private static final NereidsParser PARSER = new NereidsParser();
private String generateString(int length) {
return generateString(length, "X");
@@ -434,6 +454,198 @@ public class InsertUtilsTest {
"Cannot specify invisible column '_row_id' in INSERT
statement", error.getMessage());
}
+ @Test
+ public void writeOnlyDefaultInValuesResolvesAgainstPinnedWriterSchema() {
+ // insert into t (score, id) values (DEFAULT(score), 4): the read
schema of a table with write-only
+ // defaults has no default, so DEFAULT(score) must take the pinned
writer column's default.
+ ConnectContext ctx = MemoTestUtils.createConnectContext();
+ try {
+ PluginDrivenExternalTable table = pinWriterSchema(ctx, true);
+ Plan normalized =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("score", "id"),
+ new UnboundInlineTable(ImmutableList.of(ImmutableList.of(
+ new UnboundAlias(new Default(new
UnboundSlot("score"))),
+ new UnboundAlias(new IntegerLiteral(4)))))),
+ table, Optional.empty(), Optional.empty());
+
+ NamedExpression score = ((LogicalInlineTable)
normalized.child(0)).getConstantExprsList().get(0).get(0);
+
Assertions.assertEquals(TypeCoercionUtils.castUnbound(PARSER.parseExpression("9"),
IntegerType.INSTANCE),
+ score.child(0));
+
+ AnalysisException unknown =
Assertions.assertThrows(AnalysisException.class,
+ () ->
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("id", "score"),
+ new
UnboundInlineTable(ImmutableList.of(ImmutableList.of(
+ new UnboundAlias(new IntegerLiteral(6)),
+ new UnboundAlias(new Default(new
UnboundSlot("no_such_column"))))))),
+ table, Optional.empty(), Optional.empty()));
+
Assertions.assertTrue(unknown.getMessage().contains("no_such_column"),
unknown.getMessage());
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ @Test
+ public void writeOnlyDefaultInSelectResolvesByTargetColumnName() {
+ // DEFAULT(name) in INSERT ... SELECT names the target column whatever
the source is: here a staging
+ // relation whose own column is also called name, and a SELECT without
FROM.
+ ConnectContext ctx = MemoTestUtils.createConnectContext();
+ try {
+ PluginDrivenExternalTable table = pinWriterSchema(ctx, true);
+ Expression writeDefault = PARSER.parseExpression("'write-name'");
+
+ Plan fromStaging =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("id", "name"),
+ PARSER.parseSingle("select id, DEFAULT(name) from "
+ + "(select 7 as id, 'source-name' as name)
staging_source").child(0)),
+ table, Optional.empty(), Optional.empty());
+ LogicalProject<?> project = (LogicalProject<?>)
fromStaging.child(0);
+ Assertions.assertEquals(writeDefault,
project.getProjects().get(1).child(0));
+
+ Plan noFrom =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("id", "name"),
+ new UnboundOneRowRelation(new RelationId(1),
ImmutableList.of(
+ new UnboundAlias(new IntegerLiteral(8)),
+ new UnboundAlias(new Default(new
UnboundSlot("name")))))),
+ table, Optional.empty(), Optional.empty());
+ UnboundOneRowRelation oneRow = (UnboundOneRowRelation)
noFrom.child(0);
+ Assertions.assertEquals(writeDefault,
oneRow.getProjects().get(1).child(0));
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ @Test
+ public void writeOnlyDefaultInStarReplacementResolvesByTargetColumnName() {
+ // INSERT INTO t(id, name) SELECT src.* EXCEPT(score)
REPLACE(DEFAULT(name) AS name) FROM src: the
+ // replacement belongs to the INSERT's own select list, so
DEFAULT(name) names the target column whatever
+ // src.name's default is. UnboundStar keeps its replacements outside
its expression children.
+ ConnectContext ctx = MemoTestUtils.createConnectContext();
+ try {
+ PluginDrivenExternalTable table = pinWriterSchema(ctx, true);
+ Plan normalized =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("id", "name"),
+ PARSER.parseSingle("select src.* except(score)
replace(DEFAULT(name) as name) from src")
+ .child(0)),
+ table, Optional.empty(), Optional.empty());
+ UnboundStar star = (UnboundStar) ((LogicalProject<?>)
normalized.child(0)).getProjects().get(0);
+ Assertions.assertEquals(PARSER.parseExpression("'write-name'"),
star.getReplacedAlias().get(0).child(0));
+ Assertions.assertEquals(ImmutableList.of("src"),
star.getQualifier());
+ Assertions.assertEquals(ImmutableList.of(new
UnboundSlot("score")), star.getExceptedSlots());
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ @Test
+ public void writeOnlyDefaultInDerivedTableNamesItsOwnSource() {
+ // INSERT INTO t(score, name) SELECT v, DEFAULT(name) FROM (SELECT
DEFAULT(score) AS v FROM src) q: the
+ // derived table is a query of its own, so its DEFAULT(score) names
src.score, whose default can differ from
+ // the target's 9, and is left to binding, qualified or not. DEFAULT
in the INSERT's own select lists, set
+ // operation branches included, still names the target column.
+ ConnectContext ctx = MemoTestUtils.createConnectContext();
+ try {
+ PluginDrivenExternalTable table = pinWriterSchema(ctx, true);
+ Expression writeDefault = PARSER.parseExpression("'write-name'");
+ for (String derived : ImmutableList.of("(select DEFAULT(score) as
v from src) q",
+ "(select DEFAULT(s.score) as v from src s) q")) {
+ Plan normalized =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("score", "name"),
+ PARSER.parseSingle("select v, DEFAULT(name) from " +
derived).child(0)),
+ table, Optional.empty(), Optional.empty());
+ LogicalProject<?> outer = (LogicalProject<?>)
normalized.child(0);
+ Assertions.assertEquals(writeDefault,
outer.getProjects().get(1).child(0), derived);
+ assertDerivedDefaultLeftToBinding(outer.child(0), derived);
+ }
+
+ Plan union =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("name"),
+ PARSER.parseSingle("select DEFAULT(name) union all "
+ + "select v from (select DEFAULT(name) as v from
src) q").child(0)),
+ table, Optional.empty(), Optional.empty());
+ LogicalUnion setOperation = (LogicalUnion) union.child(0);
+ UnboundOneRowRelation oneRow = (UnboundOneRowRelation)
setOperation.child(0);
+ Assertions.assertEquals(writeDefault,
oneRow.getProjects().get(0).child(0));
+ assertDerivedDefaultLeftToBinding(setOperation.child(1).child(0),
"union branch");
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ private static void assertDerivedDefaultLeftToBinding(Plan derivedTable,
String message) {
+ Assertions.assertInstanceOf(LogicalSubQueryAlias.class, derivedTable,
message);
+ LogicalProject<?> inner = (LogicalProject<?>) derivedTable.child(0);
+ Assertions.assertInstanceOf(Default.class,
inner.getProjects().get(0).child(0), message);
+ }
+
+ @Test
+ public void defaultReferenceIsLeftToBindingWithoutWriteOnlyDefaults() {
+ // A connector whose read schema carries its defaults (hive) keeps the
existing DEFAULT(column) handling.
+ ConnectContext ctx = MemoTestUtils.createConnectContext();
+ try {
+ PluginDrivenExternalTable table = pinWriterSchema(ctx, false);
+ Plan normalized =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("id", "name"),
+ new UnboundOneRowRelation(new RelationId(1),
ImmutableList.of(
+ new UnboundAlias(new IntegerLiteral(8)),
+ new UnboundAlias(new Default(new
UnboundSlot("name")))))),
+ table, Optional.empty(), Optional.empty());
+
+ UnboundOneRowRelation oneRow = (UnboundOneRowRelation)
normalized.child(0);
+ Assertions.assertInstanceOf(Default.class,
oneRow.getProjects().get(1).child(0));
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ @Test
+ public void writeOnlyDefaultIsTypedAsItsColumnBeforeBinding() {
+ // bit_count(DEFAULT(bits)) on an INT column whose default is -1 must
count the 32 bits of an INT, not the 8
+ // of the TINYINT that -1 parses to; casting the finished value to the
INT target cannot restore them.
+ ConnectContext ctx = MemoTestUtils.createConnectContext();
+ try {
+ PluginDrivenExternalTable table = pinWriterSchema(ctx, true);
+ Expression intDefault =
TypeCoercionUtils.castUnbound(PARSER.parseExpression("-1"),
IntegerType.INSTANCE);
+
+ Expression bitCount = new UnboundFunction("bit_count",
+ ImmutableList.of(new Default(new UnboundSlot("bits"))));
+ Plan values =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("bits"),
+ new
UnboundInlineTable(ImmutableList.of(ImmutableList.of(new
UnboundAlias(bitCount))))),
+ table, Optional.empty(), Optional.empty());
+ Assertions.assertEquals(intDefault, bitCountArgument(
+ ((LogicalInlineTable)
values.child(0)).getConstantExprsList().get(0).get(0)));
+
+ Plan select =
InsertUtils.normalizePlan(connectorSink(ImmutableList.of("bits"),
+ PARSER.parseSingle("select
bit_count(DEFAULT(bits))").child(0)),
+ table, Optional.empty(), Optional.empty());
+ Assertions.assertEquals(intDefault, bitCountArgument(
+ ((UnboundOneRowRelation)
select.child(0)).getProjects().get(0)));
+ Assertions.assertEquals(IntegerType.INSTANCE,
intDefault.getDataType());
+ } finally {
+ ConnectContext.remove();
+ }
+ }
+
+ private static Expression bitCountArgument(Expression value) {
+ UnboundFunction bitCount =
value.<UnboundFunction>collectFirst(UnboundFunction.class::isInstance).get();
+ return bitCount.child(0);
+ }
+
+ private static PluginDrivenExternalTable pinWriterSchema(ConnectContext
ctx, boolean writeOnlyDefaults) {
+ PluginDrivenExternalTable table =
Mockito.mock(PluginDrivenExternalTable.class);
+ Mockito.when(table.getId()).thenReturn(17L);
+ Mockito.when(table.getName()).thenReturn("t");
+
Mockito.when(table.hasWriteOnlyDefaults()).thenReturn(writeOnlyDefaults);
+ Column id = new Column("id", PrimitiveType.INT, true);
+ id.setConnectorDefaultValueSql("NULL");
+ Column name = new Column("name", PrimitiveType.STRING, true);
+ name.setConnectorDefaultValueSql("'write-name'");
+ Column score = new Column("score", PrimitiveType.INT, true);
+ score.setConnectorDefaultValueSql("9");
+ Column bits = new Column("bits", PrimitiveType.INT, true);
+ bits.setConnectorDefaultValueSql("-1");
+ MemoTestUtils.createStatementContext(ctx, "")
+ .setConnectorWriteSchema(17L, ImmutableList.of(id, name,
score, bits));
+ return table;
+ }
+
+ private static UnboundConnectorTableSink<Plan> connectorSink(List<String>
columnNames, Plan query) {
+ return new UnboundConnectorTableSink<>(ImmutableList.of("ctl", "db",
"t"), columnNames,
+ ImmutableList.of(), ImmutableList.of(), query);
+ }
+
@Test
public void nativeExplicitInvisibleColumnRemainsSupported() {
Column id = new Column("id", PrimitiveType.INT);
diff --git
a/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
b/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
index d2945f45cf8..a5ddf7fb1a8 100644
---
a/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
+++
b/regression-test/data/external_table_p0/iceberg/test_iceberg_initial_defaults.out
@@ -387,6 +387,16 @@ missing_map_col
map<text,struct<missing_map_default_int:int>> Yes true \N
3 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
2024-12-17 23:59:59.123456+00:00
4 123e4567-e89b-12d3-a456-426614174000 0A0B0C0D 2A2B2C
2024-12-17 23:59:59.123456+00:00
+-- !parquet_default_references --
+20 35 update-where-default
+21 35 update-where-default
+22 35 merge-on-default
+
+-- !orc_default_references --
+20 35 update-where-default
+21 35 update-where-default
+22 35 merge-on-default
+
-- !parquet_branch_overwrite_default --
14 36
diff --git
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_initial_defaults.groovy
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_initial_defaults.groovy
index e03347121cf..8d4ec700a90 100644
---
a/regression-test/suites/external_table_p0/iceberg/test_iceberg_initial_defaults.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/test_iceberg_initial_defaults.groovy
@@ -600,6 +600,45 @@ suite("test_iceberg_initial_defaults", "p0,external") {
sparkUpdateMergeVerification)
}
+ // DEFAULT(column) elsewhere in a write must also resolve to the pinned
write-default (35); the read
+ // schema carries none. Rows 20-22 come from VALUES and INSERT ... SELECT
with and without FROM, then the
+ // UPDATE and MERGE predicates must match all of them. A plain SELECT
writes nothing and is rejected.
+ tableNames.each { String format, String currentTable ->
+ sql """INSERT INTO ${currentTable} (id, default_int) VALUES (20,
DEFAULT(default_int))"""
+ sql """
+ INSERT INTO ${currentTable} (id, default_int)
+ SELECT 21, DEFAULT(default_int) FROM ${currentTable} WHERE id = 7
+ """
+ sql """INSERT INTO ${currentTable} (id, default_int) SELECT 22,
DEFAULT(default_int)"""
+ sql """
+ UPDATE ${currentTable}
+ SET default_string = 'update-where-default'
+ WHERE id IN (20, 21, 22) AND default_int = DEFAULT(default_int)
+ """
+ sql """
+ MERGE INTO ${currentTable} t
+ USING (SELECT 22 AS id) s
+ ON t.id = s.id AND t.default_int = DEFAULT(t.default_int)
+ WHEN MATCHED THEN UPDATE SET default_string = 'merge-on-default'
+ """
+ "order_qt_${format}_default_references" """
+ SELECT id, default_int, default_string
+ FROM ${currentTable}
+ WHERE id IN (20, 21, 22)
+ ORDER BY id
+ """
+ }
+ test {
+ sql """SELECT DEFAULT(default_int) FROM ${parquetTable}"""
+ exception "must reference a column of the table that this statement
writes"
+ }
+ // A DELETE writes no row values, so the write-default must not decide
which rows it deletes: rows 20-22
+ // hold 35 and would all match.
+ test {
+ sql """DELETE FROM ${parquetTable} WHERE id IN (20, 21, 22) AND
default_int = DEFAULT(default_int)"""
+ exception "cannot be used in a DELETE"
+ }
+
// Pin a branch while default_int still has write-default 35, then evolve
the table-current
// schema to 36. Iceberg selects the branch as the parent snapshot but
writes the new snapshot
// with the table-current schema, so VALUES(DEFAULT) normalization and the
writer must both use
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
