[
https://issues.apache.org/jira/browse/IMPALA-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075729#comment-18075729
]
ASF subversion and git services commented on IMPALA-14589:
----------------------------------------------------------
Commit c9f8b5c0db8a2e9c35f9e59032ea0dbbcd26534f in impala's branch
refs/heads/master from Arnab Karmakar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c9f8b5c0d ]
IMPALA-14589: [Part 1] Add support for Iceberg V3 default values
Iceberg V3 introduces two types of default values to enable non-blocking
schema evolution, this patch adds support for them in Impala.
1. **initial-default**: Applied when READING old data files that were
written before a column was added to the schema. This allows adding
columns with defaults without rewriting existing data files. The
default value is materialized at read time for missing columns.
2. **write-default**: Applied when WRITING new rows that don't specify a
value for a column. This ensures consistent defaults across all
engines writing to the table, regardless of whether they're inserting
via Spark, Flink, or Impala.
How Default Values Work:
- Default values are stored in the Iceberg table's schema metadata
(not in files)
- When scanning a file missing a column, Impala checks the schema for
initial-default
- If present, the value is materialized in the template tuple before
scanning
- For writes, InsertStmt checks for write-default on unmentioned columns
- Both default types are represented as JSON-serialized literals in the schema
This patch adds support for ONLY initial-default values.
Testing:
- Updated iceberg-v3-negative.test with schema evolution expectations
- Added negative tests for complex types with default values
- Extended iceberg-v3-default-values.test with comprehensive coverage:
* Read path: SELECT, WHERE, scanner and predicate pushdown with compound
predicates
* Tests validating partition key values override initial defaults
* Time travel: Schema evolution across snapshots
Change-Id: I9f1be994a336b30b17b17819091417d777a39be9
Reviewed-on: http://gerrit.cloudera.org:8080/24088
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Iceberg V3 default values
> -------------------------
>
> Key: IMPALA-14589
> URL: https://issues.apache.org/jira/browse/IMPALA-14589
> Project: IMPALA
> Issue Type: New Feature
> Components: Backend, Frontend
> Reporter: Zoltán Borók-Nagy
> Assignee: Arnab Karmakar
> Priority: Major
> Labels: impala-iceberg
>
> Implement default value support for columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
