[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22255: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master, PR #1175. Thanks for the patch [~Rajkumar Singh]! > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.01.patch, HIVE-22255.02.patch, > HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22255: -- Status: Open (was: Patch Available) > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.01.patch, HIVE-22255.02.patch, > HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22255: -- Attachment: HIVE-22255.02.patch Status: Patch Available (was: Open) > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.01.patch, HIVE-22255.02.patch, > HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22255: -- Attachment: HIVE-22255.01.patch Status: Patch Available (was: Open) > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.01.patch, HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22255: -- Status: Open (was: Patch Available) > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajkumar Singh updated HIVE-22255: -- Attachment: HIVE-22255.patch Status: Patch Available (was: Open) > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dinesh Chitlangia updated HIVE-22255: - Summary: Hive don't trigger Major Compaction automatically if table contains only base files (was: Hive don't trigger Major Compaction automatically if table contains all base files ) > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)