morningman opened a new pull request, #69999:
URL: https://github.com/apache/doris/pull/69999
### What problem does this PR solve?
Issue Number: None
Related PR: #69299, #69488
Problem Summary:
#### In short
An Iceberg table stored in HMS and read through an HMS catalog
(`'type'='hms'`) loses every capability it inherits from the embedded Iceberg
connector in any statement that pins the table at a snapshot. Since #69299 that
is every statement that reads it: it gets no Top-N lazy materialization and no
nested column pruning, and `DEFAULT(column)` in the conditions of `UPDATE` and
`MERGE` silently matches nothing. This PR makes the snapshot overload of
`HiveConnectorMetadata#getTableSchema` pass the inherited capabilities on, as
the latest-schema overload already does. It is a master-only fix (branch-4.x
has no connector SPI), left over from #69488 (review comment 4355508100).
#### Background
fe-core resolves a table-scoped capability
(`PluginDrivenExternalTable.hasCapability`) from the catalog's connector or
from the per-table set in the table's schema. It never asks the embedded
connector that built the schema of an iceberg-on-HMS table. So
`HiveConnectorMetadata` copies the capabilities that fe-core resolves per table
(`SIBLING_INHERITABLE_CAPABILITIES`: auto and sampled analyze, Top-N lazy
materialization, nested column pruning, field-id access paths, storage
predicate pruning, nested column DDL and write-only defaults) from the owning
connector into the per-table set of each schema it delegates. Iceberg declares
seven of the eight; it does not declare sampled analyze.
It did this only in `getTableSchema(session, handle)`.
`getTableSchema(session, handle, snapshot)` returned the sibling's schema
unchanged. fe-core uses that overload for explicit time travel. Since #69299 it
also uses it for the latest-snapshot pin of every statement that reads an
Iceberg table: the pin retains its schema (`retainSchema(true)`),
`PluginDrivenMvccExternalTable` builds the schema at that pin, and
`getSchemaCacheValue()`, where `hasCapability` reads the per-table set, returns
it for the rest of the statement.
#### The problem, and what it cost
Reproduced on a local cluster with a standalone Hive 3.1.2 metastore. Two
identical format-version 3 tables `(id INT, c INT, s STRUCT<a:INT, b:STRING>)`
get two rows, then `ADD COLUMN d INT DEFAULT 42`, `RENAME COLUMN s.a TO a2` and
a third row with `d = 7`. The same statements then run against one table
through the Iceberg catalog and against the other through the HMS catalog.
| Statement | Iceberg catalog | HMS catalog, master | HMS catalog, this PR |
|---|---|---|---|
| `EXPLAIN SELECT * FROM t ORDER BY id LIMIT 1`, also with `FOR VERSION AS
OF` | `VMaterializeNode` | no lazy materialization | `VMaterializeNode` |
| `EXPLAIN SELECT struct_element(s, 'a2') FROM t` | `all access paths:
[s.a2]` | no pruning | `all access paths: [s.a2]` |
| `UPDATE t SET c = 100 WHERE d = DEFAULT(d)` | updates rows 0 and 1 |
updates nothing | updates rows 0 and 1 |
| `MERGE INTO t USING (SELECT 0 AS id) s ON t.id = s.id AND t.d =
DEFAULT(t.d) WHEN MATCHED THEN UPDATE SET c = 300` | updates row 0 | updates
nothing | updates row 0 |
| `DELETE FROM t WHERE d = DEFAULT(d)` | fails: `cannot be used in a DELETE`
| deletes nothing | fails the same way |
| `SELECT DEFAULT(d) FROM t` | fails: `must reference a column of the table
that this statement writes` | returns NULL | fails the same way |
The `UPDATE`, `MERGE` and `DELETE` rows fail silently: the statement
succeeds and changes nothing. Reads of the renamed field return the right
values in all three columns: on master the HMS catalog does not prune the
struct at all, and with this PR it prunes it with field-id access paths, as the
Iceberg catalog does. `INSERT ... VALUES (DEFAULT(d))` and `INSERT ... SELECT
DEFAULT(d)` already wrote 42 through both catalogs.
#### How this PR fixes it
Both overloads now go through one private helper, `delegatedTableSchema`. It
resolves the owning sibling once, asks it for the schema (at the pin, for the
snapshot overload), and passes the result through the existing
`reflectSiblingCapabilities`. The snapshot reaches the sibling unchanged. A
hudi-on-HMS table is unaffected: the Hudi connector declares no capabilities,
so its schema comes back as it was. A plain Hive table never reaches this
branch.
#### Results
Through the HMS catalog, every statement in the table above now plans and
answers as it does through the Iceberg catalog.
### Release note
Fix Iceberg tables read through an HMS catalog losing Top-N lazy
materialization and nested column pruning, and `DEFAULT(column)` in `UPDATE` /
`MERGE` conditions on them reading NULL instead of the column's write-default.
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- FE UT: in `HiveConnectorMetadataSiblingDelegationTest` the per-table
capability cases now check both `getTableSchema` overloads, and a new case
checks that the snapshot overload forwards the pin unchanged and resolves the
owner once. All 49 fe-connector-hive test classes pass (502 tests);
fe-connector-hive Checkstyle 0 violations. Three mutations (the snapshot
overload without the reflection, asking the sibling for its latest schema,
resolving the owner a second time) each fail only their target cases.
- Regression: new `test_iceberg_on_hms_gateway_pinned_capabilities`
(hive3 docker) runs the probe above through both catalogs; its `.out` holds
both results under paired tags. I ran it locally against a standalone Hive
3.1.2 metastore with a `file://` warehouse, with the catalog properties
adapted, not against the hive3 docker environment: without this change it fails
on the HMS catalog's first Top-N check, with it it passes.
- Manual: the table above, on a local single-BE cluster.
- Behavior changed: Yes. Statements on Iceberg tables read through an HMS
catalog plan with the capabilities they have through an Iceberg catalog, and
`DEFAULT(column)` on them resolves as it does there.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
