[
https://issues.apache.org/jira/browse/HIVE-29694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-29694.
--------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
PR has been pushed to master. Thank you for the PR [~zabetak]
[~InvisibleProgrammer]!
If you find an issue with this PR, please create a new Jira. Thank you!
> Reuse DB column descriptors when adding partitions in HMS
> ---------------------------------------------------------
>
> Key: HIVE-29694
> URL: https://issues.apache.org/jira/browse/HIVE-29694
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
> Reporter: Stamatis Zampetakis
> Assignee: Zsolt Miskolczi
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> In most cases the partitions of the table have the same schema. However,
> there are use-cases where the partitions have a different schema from the
> table. The most straightforward example comes from schema evolution where
> individual columns can be
> [added/removed/renamed|https://hive.apache.org/docs/latest/language/languagemanual-ddl/#alter-column]
> at the table level or partition level.
> {code:sql}
> create table person(id string, fname string) partitioned by (birthyear
> string);
> alter table person add partition (birthyear='1987');
> alter table person add partition (birthyear='1988');
> alter table person add partition (birthyear='1989');
> alter table person add partition (birthyear='1990');
> alter table person add partition (birthyear='1991');
> alter table person partition (birthyear='1989') add columns (lname string);
> alter table person partition (birthyear='1990') add columns (lname string);
> alter table person partition (birthyear='1991') change column fname fullname
> string;{code}
> By checking the metastore content we can see that we are using the same
> column descriptor (CD_ID = 1) for the table (SD_ID=1) and the partitions
> 1987, 1988. However, the partitions 1989, 1990, 1991 each have its own column
> descriptor. Note that partitions 1989 and 1990 have exactly the same schema
> so it should be feasible to reuse the same column descriptor.
> {noformat}
> hivedb=# SELECT "SD_ID", SUBSTRING("LOCATION", 69) AS LOCATION, "CD_ID" FROM
> "SDS";
> SD_ID | location | CD_ID
> -------+---------------------------------+-------
> 1 | warehouse/person | 1
> 2 | warehouse/person/birthyear=1987 | 1
> 3 | warehouse/person/birthyear=1988 | 1
> 4 | warehouse/person/birthyear=1989 | 2
> 5 | warehouse/person/birthyear=1990 | 3
> 6 | warehouse/person/birthyear=1991 | 4
> {noformat}
> The benefit of reusing column descriptors becomes more apparent by inspecting
> "COLUMNS_V2" table notably the descriptor 2 and 3 where the column entries
> are identical.
> {noformat}
> hivedb=# SELECT * FROM "COLUMNS_V2" ;
> CD_ID | COMMENT | COLUMN_NAME | TYPE_NAME | INTEGER_IDX
> -------+---------+-------------+-----------+-------------
> 1 | | id | string | 0
> 1 | | fname | string | 1
> 2 | | id | string | 0
> 2 | | fname | string | 1
> 2 | | lname | string | 2
> 3 | | id | string | 0
> 3 | | fname | string | 1
> 3 | | lname | string | 2
> 4 | | id | string | 0
> 4 | | fullname | string | 1
> {noformat}
> At this tiny scale the reuse benefits may be hard to see but lets assume that
> we have 1K partitions with the same schema (20 columns). Now it could
> potentially lead to 1K entries in CDS table and 20K entries in COLUMNS_V2
> table instead of 1 on each.
> The main driving motivation behind this improvement is replication scenarios
> with many tables and partitions. The replication tool calls the
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.Iface#add_partition
> for each partition on the source cluster and when the partition schema
> differs from the table schema we end up having a distinct column descriptor
> for each partition entry.
> The goal is to add an option to be able to reuse column descriptors across
> partitions of the same table. Since replication is not a dominant use-case
> the reuse logic must be behind a configuration property (disabled by default)
> to avoid performance regressions on the main use-cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
