Stamatis Zampetakis created HIVE-29694:
------------------------------------------
Summary: Reuse DB column descriptors when adding partitions in HMS
Key: HIVE-29694
URL: https://issues.apache.org/jira/browse/HIVE-29694
Project: Hive
Issue Type: Improvement
Components: Standalone Metastore
Reporter: Stamatis Zampetakis
Assignee: Stamatis Zampetakis
In most cases the partitions of the table have the same schema. However, there
are use-cases where the partitions have a different schema from the table. The
most straightforward example comes from schema evolution where individual
columns can be
[added/removed/renamed|https://hive.apache.org/docs/latest/language/languagemanual-ddl/#alter-column]
at the table level or partition level.
{code:sql}
create table person(id string, fname string) partitioned by (birthyear string);
alter table person add partition (birthyear='1987');
alter table person add partition (birthyear='1988');
alter table person add partition (birthyear='1989');
alter table person add partition (birthyear='1990');
alter table person add partition (birthyear='1991');
alter table person partition (birthyear='1989') add columns (lname string);
alter table person partition (birthyear='1990') add columns (lname string);
alter table person partition (birthyear='1991') change column fname fullname
string;{code}
By checking the metastore content we can see that we are using the same column
descriptor (CD_ID = 1) for the table (SD_ID=1) and the partitions 1987, 1988.
However, the partitions 1989, 1990, 1991 each have its own column descriptor.
Note that partitions 1989 and 1990 have exactly the same schema so it should be
feasible to reuse the same column descriptor.
{noformat}
hivedb=# SELECT "SD_ID", SUBSTRING("LOCATION", 69) AS LOCATION, "CD_ID" FROM
"SDS";
SD_ID | location | CD_ID
-------+---------------------------------+-------
1 | warehouse/person | 1
2 | warehouse/person/birthyear=1987 | 1
3 | warehouse/person/birthyear=1988 | 1
4 | warehouse/person/birthyear=1989 | 2
5 | warehouse/person/birthyear=1990 | 3
6 | warehouse/person/birthyear=1991 | 4
{noformat}
The benefit of reusing column descriptors becomes more apparent by inspecting
"COLUMNS_V2" table notably the descriptor 2 and 3 where the column entries are
identical.
{noformat}
hivedb=# SELECT * FROM "COLUMNS_V2" ;
CD_ID | COMMENT | COLUMN_NAME | TYPE_NAME | INTEGER_IDX
-------+---------+-------------+-----------+-------------
1 | | id | string | 0
1 | | fname | string | 1
2 | | id | string | 0
2 | | fname | string | 1
2 | | lname | string | 2
3 | | id | string | 0
3 | | fname | string | 1
3 | | lname | string | 2
4 | | id | string | 0
4 | | fullname | string | 1
{noformat}
At this tiny scale the reuse benefits may be hard to see but lets assume that
we have 1K partitions with the same schema (20 columns). Now it could
potentially lead to 1K entries in CDS table and 20K entries in COLUMNS_V2 table
instead of 1 on each.
The main driving motivation behind this improvement is replication scenarios
with many tables and partitions. The replication tool calls the
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore.Iface#add_partition
for each partition on the source cluster and when the partition schema differs
from the table schema we end up having a distinct column descriptor for each
partition entry.
The goal is to add an option to be able to reuse column descriptors across
partitions of the same table. Since replication is not a dominant use-case the
reuse logic must be behind a configuration property (disabled by default) to
avoid performance regressions on the main use-cases.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
