[
https://issues.apache.org/jira/browse/HIVE-30147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryu Kobayashi updated HIVE-30147:
---------------------------------
Description:
h2. Summary
The Iceberg copy-on-write MERGE rewrite does not fully preserve the semantics
of the native MERGE implementation.
The affected areas include NULL-key handling, source-only row detection, SQL
three-valued predicate evaluation, `WHEN MATCHED` clause ordering, and
cardinality-violation handling. These issues should be corrected so that
copy-on-write MERGE behaves consistently with native MERGE.
h2. Root cause
The current implementation in `CopyOnWriteMergeRewriter` rewrites MERGE
statements into a source CTE and multiple `UNION ALL` branches.
Several parts of the rewrite rely on derived predicates and string-level
predicate composition:
- `MergeSemanticAnalyzer.OnClauseAnalyzer` derives source-only predicates from
NULL checks on target join columns.
- `CopyOnWriteMergeRewriter` reuses those predicates for the INSERT and
carry-over branches.
- Target/source presence is not represented by an explicit join-side marker
such as the target file path.
- Multiple predicates are concatenated with `AND` and `OR`, which can change
their effective precedence when the original expressions contain OR operators.
- Later `WHEN MATCHED` clauses are gated with `NOT(previousPredicate)`, which
does not implement SQL `IS NOT TRUE` semantics when the previous predicate
evaluates to `UNKNOWN`.
- The copy-on-write rewrite does not currently use the standard
cardinality-violation branch implemented by `MergeRewriter`.
h2. Repeated source evaluation and duplicated execution work
The current CoW rewrite also evaluates the same source relation and match
predicate multiple times. The marker CTE contains separate representative-row
and grouped-count scans, and the cardinality-validation branch independently
evaluates the MERGE source relation.
This can produce inconsistent results for non-deterministic or stateful
predicates such as `rand()`, because the scans may observe different row sets.
It also adds a second full scan, aggregation, join, and shuffle to CoW DML, and
can duplicate the work for MERGE source relations.
The follow-up design should evaluate the source relation and match disposition
once through a shared execution stream, while preserving native MERGE semantics
and avoiding duplicated window/aggregation pipelines.
h2. Confirmed / observed behavior
The following cases require regression coverage and correction:
1. A target row whose join key is NULL can be mistaken for a source-only row
when another row in the same data file is matched and the file is rewritten.
2. A target-only row with NULL join keys can satisfy the derived source-only
predicate and produce an unintended INSERT.
3. A matched predicate that evaluates to `UNKNOWN` can prevent a later `WHEN
MATCHED` clause from executing.
4. Unparenthesized combinations of `ON` predicates and clause predicates can
cause source-only rows or unrelated target rows to enter the replacement-marker
path.
5. Multiple source rows matching the same target row can produce duplicate CoW
updates or incorrect affected-row behavior instead of the standard cardinality
violation.
6. On constrained targets, the full-width CoW cardinality branch may interact
with NOT NULL/CHECK enforcement before the intended cardinality error is
reported. This requires explicit validation with both CBO enabled and disabled.
h2. Impact / exposure
These issues affect the correctness and compatibility of Iceberg copy-on-write
MERGE statements.
Potential effects include:
- Silent loss of legitimate target rows.
- Unintended insertion of NULL or source-derived rows.
- Different results from native MERGE for predicates involving NULL or OR
expressions.
- Duplicate or nondeterministic updates when source cardinality is invalid.
- Cardinality errors being masked by constraint violations.
- Inconsistent behavior depending on CBO configuration.
This issue is separate from HIVE-30064's affected-row reporting scope.
HIVE-30064 tracks logical affected-row counts for CoW DML; this issue tracks
the broader correctness and semantic alignment of CoW MERGE rewriting.
The implementation should preserve native MERGE semantics and include
regression coverage for the affected cases.
was:
h2. Summary
The Iceberg copy-on-write MERGE rewrite does not fully preserve the semantics
of the native MERGE implementation.
The affected areas include NULL-key handling, source-only row detection, SQL
three-valued predicate evaluation, `WHEN MATCHED` clause ordering, and
cardinality-violation handling. These issues should be corrected so that
copy-on-write MERGE behaves consistently with native MERGE.
h2. Root cause
The current implementation in `CopyOnWriteMergeRewriter` rewrites MERGE
statements into a source CTE and multiple `UNION ALL` branches.
Several parts of the rewrite rely on derived predicates and string-level
predicate composition:
- `MergeSemanticAnalyzer.OnClauseAnalyzer` derives source-only predicates from
NULL checks on target join columns.
- `CopyOnWriteMergeRewriter` reuses those predicates for the INSERT and
carry-over branches.
- Target/source presence is not represented by an explicit join-side marker
such as the target file path.
- Multiple predicates are concatenated with `AND` and `OR`, which can change
their effective precedence when the original expressions contain OR operators.
- Later `WHEN MATCHED` clauses are gated with `NOT(previousPredicate)`, which
does not implement SQL `IS NOT TRUE` semantics when the previous predicate
evaluates to `UNKNOWN`.
- The copy-on-write rewrite does not currently use the standard
cardinality-violation branch implemented by `MergeRewriter`.
h2. Confirmed / observed behavior
The following cases require regression coverage and correction:
1. A target row whose join key is NULL can be mistaken for a source-only row
when another row in the same data file is matched and the file is rewritten.
2. A target-only row with NULL join keys can satisfy the derived source-only
predicate and produce an unintended INSERT.
3. A matched predicate that evaluates to `UNKNOWN` can prevent a later `WHEN
MATCHED` clause from executing.
4. Unparenthesized combinations of `ON` predicates and clause predicates can
cause source-only rows or unrelated target rows to enter the replacement-marker
path.
5. Multiple source rows matching the same target row can produce duplicate CoW
updates or incorrect affected-row behavior instead of the standard cardinality
violation.
6. On constrained targets, the full-width CoW cardinality branch may interact
with NOT NULL/CHECK enforcement before the intended cardinality error is
reported. This requires explicit validation with both CBO enabled and disabled.
h2. Impact / exposure
These issues affect the correctness and compatibility of Iceberg copy-on-write
MERGE statements.
Potential effects include:
- Silent loss of legitimate target rows.
- Unintended insertion of NULL or source-derived rows.
- Different results from native MERGE for predicates involving NULL or OR
expressions.
- Duplicate or nondeterministic updates when source cardinality is invalid.
- Cardinality errors being masked by constraint violations.
- Inconsistent behavior depending on CBO configuration.
This issue is separate from HIVE-30064's affected-row reporting scope.
HIVE-30064 tracks logical affected-row counts for CoW DML; this issue tracks
the broader correctness and semantic alignment of CoW MERGE rewriting.
The implementation should preserve native MERGE semantics and include
regression coverage for the affected cases.
> Align Iceberg copy-on-write MERGE semantics with native MERGE
> -------------------------------------------------------------
>
> Key: HIVE-30147
> URL: https://issues.apache.org/jira/browse/HIVE-30147
> Project: Hive
> Issue Type: Bug
> Reporter: Ryu Kobayashi
> Assignee: Ryu Kobayashi
> Priority: Major
>
> h2. Summary
> The Iceberg copy-on-write MERGE rewrite does not fully preserve the semantics
> of the native MERGE implementation.
> The affected areas include NULL-key handling, source-only row detection, SQL
> three-valued predicate evaluation, `WHEN MATCHED` clause ordering, and
> cardinality-violation handling. These issues should be corrected so that
> copy-on-write MERGE behaves consistently with native MERGE.
> h2. Root cause
> The current implementation in `CopyOnWriteMergeRewriter` rewrites MERGE
> statements into a source CTE and multiple `UNION ALL` branches.
> Several parts of the rewrite rely on derived predicates and string-level
> predicate composition:
> - `MergeSemanticAnalyzer.OnClauseAnalyzer` derives source-only predicates
> from NULL checks on target join columns.
> - `CopyOnWriteMergeRewriter` reuses those predicates for the INSERT and
> carry-over branches.
> - Target/source presence is not represented by an explicit join-side marker
> such as the target file path.
> - Multiple predicates are concatenated with `AND` and `OR`, which can change
> their effective precedence when the original expressions contain OR operators.
> - Later `WHEN MATCHED` clauses are gated with `NOT(previousPredicate)`,
> which does not implement SQL `IS NOT TRUE` semantics when the previous
> predicate evaluates to `UNKNOWN`.
> - The copy-on-write rewrite does not currently use the standard
> cardinality-violation branch implemented by `MergeRewriter`.
> h2. Repeated source evaluation and duplicated execution work
> The current CoW rewrite also evaluates the same source relation and match
> predicate multiple times. The marker CTE contains separate representative-row
> and grouped-count scans, and the cardinality-validation branch independently
> evaluates the MERGE source relation.
> This can produce inconsistent results for non-deterministic or stateful
> predicates such as `rand()`, because the scans may observe different row
> sets. It also adds a second full scan, aggregation, join, and shuffle to CoW
> DML, and can duplicate the work for MERGE source relations.
> The follow-up design should evaluate the source relation and match
> disposition once through a shared execution stream, while preserving native
> MERGE semantics and avoiding duplicated window/aggregation pipelines.
> h2. Confirmed / observed behavior
> The following cases require regression coverage and correction:
> 1. A target row whose join key is NULL can be mistaken for a source-only row
> when another row in the same data file is matched and the file is rewritten.
> 2. A target-only row with NULL join keys can satisfy the derived source-only
> predicate and produce an unintended INSERT.
> 3. A matched predicate that evaluates to `UNKNOWN` can prevent a later `WHEN
> MATCHED` clause from executing.
> 4. Unparenthesized combinations of `ON` predicates and clause predicates can
> cause source-only rows or unrelated target rows to enter the
> replacement-marker path.
> 5. Multiple source rows matching the same target row can produce duplicate
> CoW updates or incorrect affected-row behavior instead of the standard
> cardinality violation.
> 6. On constrained targets, the full-width CoW cardinality branch may interact
> with NOT NULL/CHECK enforcement before the intended cardinality error is
> reported. This requires explicit validation with both CBO enabled and
> disabled.
> h2. Impact / exposure
> These issues affect the correctness and compatibility of Iceberg
> copy-on-write MERGE statements.
> Potential effects include:
> - Silent loss of legitimate target rows.
> - Unintended insertion of NULL or source-derived rows.
> - Different results from native MERGE for predicates involving NULL or OR
> expressions.
> - Duplicate or nondeterministic updates when source cardinality is invalid.
> - Cardinality errors being masked by constraint violations.
> - Inconsistent behavior depending on CBO configuration.
> This issue is separate from HIVE-30064's affected-row reporting scope.
> HIVE-30064 tracks logical affected-row counts for CoW DML; this issue tracks
> the broader correctness and semantic alignment of CoW MERGE rewriting.
> The implementation should preserve native MERGE semantics and include
> regression coverage for the affected cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
