Vaibhav Gumashta created HIVE-21167:
---------------------------------------
Summary: Bucketing: Bucketing version 1 is incorrectly
partitioning data
Key: HIVE-21167
URL: https://issues.apache.org/jira/browse/HIVE-21167
Project: Hive
Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Vaibhav Gumashta
Using murmur hash for bucketing columns was introduced in HIVE-18910, following
which {{'bucketing_version'='1'}} stands for the old behaviour (where for
example integer columns were partitioned based on mod values). Looks like we
have a bug in the old bucketing scheme now. I could repro it when modified the
existing schema using an alter table add column and adding new data. Repro:
{code}
0: jdbc:hive2://localhost:10010> create transactional table acid_ptn_bucket1 (a
int, b int) partitioned by(ds string) clustered by (a) into 2 buckets stored as
ORC TBLPROPERTIES('bucketing_version'='1', 'transactional'='true',
'transactional_properties'='default');
No rows affected (0.418 seconds)
0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
values(1,2,'today'),(1,3,'today'),(1,4,'yesterday'),(2,2,'yesterday'),(2,3,'today'),(2,4,'today');
6 rows affected (3.695 seconds)
{code}
Data from ORC file (data as expected):
{code}
/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00000
{"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 0,
"currentTransaction": 1, "row": {"a": 2, "b": 4}}
{"operation": 0, "originalTransaction": 1, "bucket": 536870912, "rowId": 1,
"currentTransaction": 1, "row": {"a": 2, "b": 3}}
/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000001_0000001_0000/bucket_00001
{"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 0,
"currentTransaction": 1, "row": {"a": 1, "b": 3}}
{"operation": 0, "originalTransaction": 1, "bucket": 536936448, "rowId": 1,
"currentTransaction": 1, "row": {"a": 1, "b": 2}}
{code}
Modifying table schema and inserting new data:
{code}
0: jdbc:hive2://localhost:10010> alter table acid_ptn_bucket1 add columns(c
int);
No rows affected (0.541 seconds)
0: jdbc:hive2://localhost:10010> insert into acid_ptn_bucket1 partition (ds)
values(3,2,1000,'yesterday'),(3,3,1001,'today'),(3,4,1002,'yesterday'),(4,2,1003,'today'),
(4,3,1004,'yesterday'),(4,4,1005,'today');
6 rows affected (3.699 seconds)
{code}
Data from ORC file (wrong partitioning):
{code}
/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00000
{"operation": 0, "originalTransaction": 3, "bucket": 536870912, "rowId": 0,
"currentTransaction": 3, "row": {"a": 3, "b": 3, "c": 1001}}
/apps/hive/warehouse/acid_ptn_bucket1/ds=today/delta_0000003_0000003_0000/bucket_00001
{"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 0,
"currentTransaction": 3, "row": {"a": 4, "b": 4, "c": 1005}}
{"operation": 0, "originalTransaction": 3, "bucket": 536936448, "rowId": 1,
"currentTransaction": 3, "row": {"a": 4, "b": 2, "c": 1003}}
{code}
As seen above, the expected behaviour is that new data with column 'a' being 3
should go to bucket1 and column 'a' being 4 should go to bucket0, but the
partitioning is wrong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
