[jira] [Commented] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029699#comment-17029699 ] Laszlo Pinter commented on HIVE-22818: -- @Thanks [~pvary] for review & commit! > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22818.01.patch, HIVE-22818.02.patch, > HIVE-22818.03.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Attachment: HIVE-22818.03.patch > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch, HIVE-22818.02.patch, > HIVE-22818.03.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Attachment: HIVE-22818.02.patch > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch, HIVE-22818.02.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17028807#comment-17028807 ] Laszlo Pinter commented on HIVE-22729: -- Thank you [~pvary], [~klcopp] [~dkuzmenko]! > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch, > HIVE-22729.12.patch, HIVE-22729.13.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Attachment: HIVE-22818.01.patch > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22818: - Status: Patch Available (was: Open) > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22818.01.patch > > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22818) Preparation for jetty 9.4.26 upgrade
[ https://issues.apache.org/jira/browse/HIVE-22818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22818: > Preparation for jetty 9.4.26 upgrade > > > Key: HIVE-22818 > URL: https://issues.apache.org/jira/browse/HIVE-22818 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > > Make some code adjustment, before upgrading jetty to 9.4.26. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.13.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch, > HIVE-22729.12.patch, HIVE-22729.13.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.12.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch, > HIVE-22729.12.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.11.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch, HIVE-22729.10.patch, HIVE-22729.11.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026641#comment-17026641 ] Laszlo Pinter commented on HIVE-21487: -- I created a couple of performance tests using the HMS benchmarking tool, and I noticed a slight performance improvement on Initiator side with new indexes on COMPLETED_COMPACTIONS: (cc_database, cc_table, cc_partition). [~pvary] [~dkuzmenko] [~klcopp] Could you please review? Thanks > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch > > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.10.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch, HIVE-22729.10.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22793) Update default settings in HMS Benchmarking tool
[ https://issues.apache.org/jira/browse/HIVE-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22793: - Status: Patch Available (was: Open) > Update default settings in HMS Benchmarking tool > > > Key: HIVE-22793 > URL: https://issues.apache.org/jira/browse/HIVE-22793 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Trivial > Attachments: HIVE-22793.01.patch > > > HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22793) Update default settings in HMS Benchmarking tool
[ https://issues.apache.org/jira/browse/HIVE-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22793: - Attachment: HIVE-22793.01.patch > Update default settings in HMS Benchmarking tool > > > Key: HIVE-22793 > URL: https://issues.apache.org/jira/browse/HIVE-22793 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Trivial > Attachments: HIVE-22793.01.patch > > > HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22793) Update default settings in HMS Benchmarking tool
[ https://issues.apache.org/jira/browse/HIVE-22793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22793: > Update default settings in HMS Benchmarking tool > > > Key: HIVE-22793 > URL: https://issues.apache.org/jira/browse/HIVE-22793 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Trivial > > HMS Benchmarking tool has invalid default setting values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17025736#comment-17025736 ] Laszlo Pinter commented on HIVE-22729: -- [~pvary] [~dkuzmenko] [~klcopp] Can I get a +1 from you? Thank you > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.09.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch, > HIVE-22729.09.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.08.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch, HIVE-22729.08.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22765) Quote table names in CompactionTxnHandler
[ https://issues.apache.org/jira/browse/HIVE-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22765: > Quote table names in CompactionTxnHandler > - > > Key: HIVE-22765 > URL: https://issues.apache.org/jira/browse/HIVE-22765 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Zoltan Chovan >Priority: Major > > When I run a metastore backed up by a postgres db, I get the following > exception: > {code:java} > MetaException(message:Unable to connect to transaction database > org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does > not existMetaException(message:Unable to connect to transaction database > org.postgresql.util.PSQLException: ERROR: relation "compaction_queue" does > not exist Position: 83 at > org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2468) > at > org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2211) > at > org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:309) > at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446) at > org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370) at > org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:311) at > org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:297) at > org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:274) at > org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:225) at > org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108) > at > org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java) > at > org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findNextToCompact(CompactionTxnHandler.java:157) > at > org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.find_next_compact(HiveMetaStore.java:8061) > at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) > at > org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) > at com.sun.proxy.$Proxy28.find_next_compact(Unknown Source) at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.findNextCompact(HiveMetaStoreClient.java:4046) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:208) > at com.sun.proxy.$Proxy29.findNextCompact(Unknown Source) at > org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:102){code} > This is due to missing quotes around table names. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21487: - Attachment: HIVE-21847.02.patch > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21847.01.patch, HIVE-21847.02.patch > > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22764) Create new command for "optimize" compaction and have basic implementation.
[ https://issues.apache.org/jira/browse/HIVE-22764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021905#comment-17021905 ] Laszlo Pinter commented on HIVE-22764: -- [~aditya-shah] Please provide a review board/pull request link to make the review easier. > Create new command for "optimize" compaction and have basic implementation. > --- > > Key: HIVE-22764 > URL: https://issues.apache.org/jira/browse/HIVE-22764 > Project: Hive > Issue Type: Sub-task >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22764.patch > > > Created new blocking compaction (added compaction type "optimize") by adding > a lock request on the compaction's transaction. It works mostly like > mmMajorCompaction and writes files w/o row_IDs. I have added an additional > table property to provide optimize columns that is used by the compactor to > cluster the data by. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.07.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch, HIVE-22729.07.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reopened HIVE-22126: -- Assignee: Eugene Chung (was: Laszlo Pinter) > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22126) hive-exec packaging should shade guava
[ https://issues.apache.org/jira/browse/HIVE-22126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17021223#comment-17021223 ] Laszlo Pinter commented on HIVE-22126: -- [~euigeun_chung] feel free to reopen and take over the jira. > hive-exec packaging should shade guava > -- > > Key: HIVE-22126 > URL: https://issues.apache.org/jira/browse/HIVE-22126 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22126.01.patch > > > The ql/pom.xml includes complete guava library into hive-exec.jar > https://github.com/apache/hive/blob/master/ql/pom.xml#L990 This causes a > problems for downstream clients of hive which have hive-exec.jar in their > classpath since they are pinned to the same guava version as that of hive. > We should shade guava classes so that other components which depend on > hive-exec can independently use a different version of guava as needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.06.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch, > HIVE-22729.06.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22255) Hive don't trigger Major Compaction automatically if table contains only base files
[ https://issues.apache.org/jira/browse/HIVE-22255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020240#comment-17020240 ] Laszlo Pinter commented on HIVE-22255: -- [~Rajkumar Singh] Could you please provide a review board/pull request where I can post my remarks? Thanks. > Hive don't trigger Major Compaction automatically if table contains only base > files > > > Key: HIVE-22255 > URL: https://issues.apache.org/jira/browse/HIVE-22255 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 3.1.2 > Environment: Hive-3.1.1 >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-22255.01.patch, HIVE-22255.patch > > > user may run into the issue if the table consists of all base files but no > delta, then the following condition will yield false and automatic major > compaction will be skipped. > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L313] > > Steps to Reproduce: > # create Acid table > {code:java} > // create table myacid(id int); > {code} > # Run multiple insert table > {code:java} > // insert overwrite table myacid values(1);insert overwrite table myacid > values(2),(3),(4){code} > # DFS ls output > {code:java} > // dfs -ls -R /warehouse/tablespace/managed/hive/myacid; > ++ > | DFS Output | > ++ > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 610 2019-09-27 16:42 > /warehouse/tablespace/managed/hive/myacid/base_001/bucket_0 | > | drwxrwx---+ - hive hadoop 0 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002 | > | -rw-rw+ 3 hive hadoop 1 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/_orc_acid_version | > | -rw-rw+ 3 hive hadoop 633 2019-09-27 16:43 > /warehouse/tablespace/managed/hive/myacid/base_002/bucket_0 | > ++{code} > > you will see that Major compaction will not be trigger until you run alter > table compact MAJOR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.05.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch, HIVE-22729.05.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21487: - Status: Patch Available (was: Open) > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21847.01.patch > > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21487: - Attachment: HIVE-21847.01.patch > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21847.01.patch > > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.04.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch, HIVE-22729.04.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.03.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: (was: HIVE-22729.03.patch) > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.03.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch, > HIVE-22729.03.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.02.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch, HIVE-22729.02.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019560#comment-17019560 ] Laszlo Pinter commented on HIVE-21487: -- We should add the following indexes to compaction related tables: COMPACTION_QUEUE: cq_id and (cq_state, cq_id) COMPLETED_COMPACTIONS: (cc_database, cc_table, cc_partition) > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21487) COMPLETED_COMPACTIONS table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-21487: Assignee: Laszlo Pinter > COMPLETED_COMPACTIONS table missing appropriate indexes > --- > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21487) COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes
[ https://issues.apache.org/jira/browse/HIVE-21487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21487: - Summary: COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes (was: COMPLETED_COMPACTIONS table missing appropriate indexes) > COMPLETED_COMPACTIONS and COMPACTION_QUEUE table missing appropriate indexes > > > Key: HIVE-21487 > URL: https://issues.apache.org/jira/browse/HIVE-21487 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Todd Lipcon >Assignee: Laszlo Pinter >Priority: Major > > Looking at a MySQL install where HMS is pointed on Hive 3.1, I see a constant > stream of queries of the form: > {code} > select CC_STATE from COMPLETED_COMPACTIONS where CC_DATABASE = > 'tpcds_orc_exact_1000' and CC_TABLE = 'catalog_returns' and CC_PARTITION = > 'cr_returned_date_sk=2452851' and CC_STATE != 'a' order by CC_ID desc; > {code} > but the COMPLETED_COMPACTIONS table has no index. In this case it's resulting > in a full table scan over 115k rows, which takes around 100ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[ https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019534#comment-17019534 ] Laszlo Pinter commented on HIVE-21931: -- [~csringhofer] It's a session property. > Slow compaction for tiny tables > --- > > Key: HIVE-21931 > URL: https://issues.apache.org/jira/browse/HIVE-21931 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Csaba Ringhofer >Priority: Major > Labels: compaction > > I observed the issue in Impala development environment when (major) > compacting insert_only transactional tables in Hive. The compaction could > take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The > actual work was done much earlier, the new base file was correctly written to > HDFS, and Hive seemed to wait without doing any work. > The compactions are started manually, hive.compactor.initiator.on=false to > avoid "surprise compaction" during tests. > {code} > hive.compactor.abortedtxn.threshold=1000 > hive.compactor.check.interval=300s > hive.compactor.cleaner.run.interval=5000ms > hive.compactor.compact.insert.only=true > hive.compactor.crud.query.based=false > hive.compactor.delta.num.threshold=10 > hive.compactor.delta.pct.threshold=0.1 > hive.compactor.history.reaper.interval=2m > hive.compactor.history.retention.attempted=2 > hive.compactor.history.retention.failed=3 > hive.compactor.history.retention.succeeded=3 > hive.compactor.initiator.failed.compacts.threshold=2 > hive.compactor.initiator.on=false > hive.compactor.max.num.delta=500 > hive.compactor.worker.threads=4 > hive.compactor.worker.timeout=86400s > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[ https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019501#comment-17019501 ] Laszlo Pinter commented on HIVE-21931: -- [~csringhofer] With HIVE-22554 you can configure the initial wait time out to a much lower value, like 2000 millisec. Does that satisfies your needs? > Slow compaction for tiny tables > --- > > Key: HIVE-21931 > URL: https://issues.apache.org/jira/browse/HIVE-21931 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Csaba Ringhofer >Priority: Major > Labels: compaction > > I observed the issue in Impala development environment when (major) > compacting insert_only transactional tables in Hive. The compaction could > take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The > actual work was done much earlier, the new base file was correctly written to > HDFS, and Hive seemed to wait without doing any work. > The compactions are started manually, hive.compactor.initiator.on=false to > avoid "surprise compaction" during tests. > {code} > hive.compactor.abortedtxn.threshold=1000 > hive.compactor.check.interval=300s > hive.compactor.cleaner.run.interval=5000ms > hive.compactor.compact.insert.only=true > hive.compactor.crud.query.based=false > hive.compactor.delta.num.threshold=10 > hive.compactor.delta.pct.threshold=0.1 > hive.compactor.history.reaper.interval=2m > hive.compactor.history.retention.attempted=2 > hive.compactor.history.retention.failed=3 > hive.compactor.history.retention.succeeded=3 > hive.compactor.initiator.failed.compacts.threshold=2 > hive.compactor.initiator.on=false > hive.compactor.max.num.delta=500 > hive.compactor.worker.threads=4 > hive.compactor.worker.timeout=86400s > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21931) Slow compaction for tiny tables
[ https://issues.apache.org/jira/browse/HIVE-21931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019458#comment-17019458 ] Laszlo Pinter commented on HIVE-21931: -- [~csringhofer] Did you run compaction in blocking mode? HIVE-22554 provides a way to configure the wait time out. > Slow compaction for tiny tables > --- > > Key: HIVE-21931 > URL: https://issues.apache.org/jira/browse/HIVE-21931 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Csaba Ringhofer >Priority: Major > Labels: compaction > > I observed the issue in Impala development environment when (major) > compacting insert_only transactional tables in Hive. The compaction could > take ~10 minutes even when it only had to merge 2 rows from 2 inserts. The > actual work was done much earlier, the new base file was correctly written to > HDFS, and Hive seemed to wait without doing any work. > The compactions are started manually, hive.compactor.initiator.on=false to > avoid "surprise compaction" during tests. > {code} > hive.compactor.abortedtxn.threshold=1000 > hive.compactor.check.interval=300s > hive.compactor.cleaner.run.interval=5000ms > hive.compactor.compact.insert.only=true > hive.compactor.crud.query.based=false > hive.compactor.delta.num.threshold=10 > hive.compactor.delta.pct.threshold=0.1 > hive.compactor.history.reaper.interval=2m > hive.compactor.history.retention.attempted=2 > hive.compactor.history.retention.failed=3 > hive.compactor.history.retention.succeeded=3 > hive.compactor.initiator.failed.compacts.threshold=2 > hive.compactor.initiator.on=false > hive.compactor.max.num.delta=500 > hive.compactor.worker.threads=4 > hive.compactor.worker.timeout=86400s > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Status: Patch Available (was: Open) > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22729: - Attachment: HIVE-22729.01.patch > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22729.01.patch > > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019340#comment-17019340 ] Laszlo Pinter commented on HIVE-22703: -- Thanks [~pvary] and [~dkuzmenko] > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22705) LLAP cache is polluted by query-based compactor
[ https://issues.apache.org/jira/browse/HIVE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017764#comment-17017764 ] Laszlo Pinter commented on HIVE-22705: -- +1 (non-binding and pending tests) > LLAP cache is polluted by query-based compactor > --- > > Key: HIVE-22705 > URL: https://issues.apache.org/jira/browse/HIVE-22705 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22705.0.patch > > > One of the steps that query-based compaction does is the verification of ACID > sort order by using the _validate_acid_sort_order_ UDF. This is a > prerequisite before the actual compaction can happen, and is done by a [query > that reads the whole table > content|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/MajorQueryCompactor.java#L161-L167]. > This results in the whole table content being populated into the cache. The > problem is that this content is not useful and will rather pollute the cache > space, as it can never be used again: cache content binds to files (file IDs) > that obviously will be changed in this case by compaction. > I propose we disable LLAP caching in the session of query-based compaction's > queries. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22729) Provide a failure reason for failed compactions
[ https://issues.apache.org/jira/browse/HIVE-22729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22729: > Provide a failure reason for failed compactions > --- > > Key: HIVE-22729 > URL: https://issues.apache.org/jira/browse/HIVE-22729 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > We should provide a compaction failure reason as easily accessible as > possible. Like in the result of the {{SHOW COMPACTIONS}} command. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Attachment: HIVE-22703.02.patch > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22703.01.patch, HIVE-22703.02.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Status: Patch Available (was: Open) > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22703.01.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Attachment: HIVE-22703.01.patch > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22703.01.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22703: - Priority: Minor (was: Major) > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22703.01.patch > > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.14.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch, > HIVE-20934.12.patch, HIVE-20934.13.patch, HIVE-20934.14.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.13.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch, > HIVE-20934.12.patch, HIVE-20934.13.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.12.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch, > HIVE-20934.12.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.11.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch, HIVE-20934.11.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.10.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch, HIVE-20934.10.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.09.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch, > HIVE-20934.09.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22703) Compaction configuration check when starting HMS/HS2
[ https://issues.apache.org/jira/browse/HIVE-22703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22703: > Compaction configuration check when starting HMS/HS2 > > > Key: HIVE-22703 > URL: https://issues.apache.org/jira/browse/HIVE-22703 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > Currently when starting HMS we can have bugous configuration which prevents > compatction to work. We should find a way to inform the admin about the > configuration error, or even prevent HMS to start in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.08.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch, HIVE-20934.08.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.07.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch, HIVE-20934.07.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.06.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch, > HIVE-20934.06.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.05.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch, HIVE-20934.05.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22700) Compactions may leak memory when unauthorized
[ https://issues.apache.org/jira/browse/HIVE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22700: - Attachment: HIVE-22700.01.patch > Compactions may leak memory when unauthorized > - > > Key: HIVE-22700 > URL: https://issues.apache.org/jira/browse/HIVE-22700 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22700.01.patch > > > Initiator class determines compaction type periodically. Initiator either > runs as hive user or impersonates the owner of the table. When impersonation > is used, Initiator#checkForCompaction may leak memory. If impersonation > (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore > does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22700) Compactions may leak memory when unauthorized
[ https://issues.apache.org/jira/browse/HIVE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22700: - Status: Patch Available (was: Open) > Compactions may leak memory when unauthorized > - > > Key: HIVE-22700 > URL: https://issues.apache.org/jira/browse/HIVE-22700 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22700.01.patch > > > Initiator class determines compaction type periodically. Initiator either > runs as hive user or impersonates the owner of the table. When impersonation > is used, Initiator#checkForCompaction may leak memory. If impersonation > (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore > does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22700) Compactions may leak memory when unauthorized
[ https://issues.apache.org/jira/browse/HIVE-22700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22700: > Compactions may leak memory when unauthorized > - > > Key: HIVE-22700 > URL: https://issues.apache.org/jira/browse/HIVE-22700 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > Initiator class determines compaction type periodically. Initiator either > runs as hive user or impersonates the owner of the table. When impersonation > is used, Initiator#checkForCompaction may leak memory. If impersonation > (ugi.doAs()) call fails, FileSystem.closeAllForUGI does not run, therefore > does not clean the file system cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22661) Compaction fails on non bucketed table with data loaded inpath
[ https://issues.apache.org/jira/browse/HIVE-22661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17009540#comment-17009540 ] Laszlo Pinter commented on HIVE-22661: -- [~szita] Thanks for the patch. +1 (non-binding) > Compaction fails on non bucketed table with data loaded inpath > -- > > Key: HIVE-22661 > URL: https://issues.apache.org/jira/browse/HIVE-22661 > Project: Hive > Issue Type: Bug >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22661.0.patch, HIVE-22661.1.patch, > HIVE-22661.2.patch > > > Compaction cannot handle situations where: > * data was ingested with {{LOAD DATA INPATH}} > * this ingest method is run multiple times, and > ** with different number of files getting created in the delta directories > Therefore, for file/dir structures such as: > {code:java} > /warehouse/tablespace/managed/hive/comp3/delta_001_001_ > /warehouse/tablespace/managed/hive/comp3/delta_001_001_/00_0 > /warehouse/tablespace/managed/hive/comp3/delta_001_001_/01_0 > /warehouse/tablespace/managed/hive/comp3/delta_002_002_ > /warehouse/tablespace/managed/hive/comp3/delta_002_002_/00_0 > /warehouse/tablespace/managed/hive/comp3/delta_002_002_/01_0 > /warehouse/tablespace/managed/hive/comp3/delta_002_002_/02_0 > {code} > Although the table is not bucketed, bucket is calculated from the (raw) > files' names. Compaction in the above case will fail on delta1-1 not having > data for 'bucket' 2. > Steps to repro using small dataset: > {code:java} > set tez.grouping.min-size=8; > set tez.grouping.max-size=8; > set mapreduce.input.fileinputformat.split.minsize=8; > set mapreduce.input.fileinputformat.split.minsize=8; > create external table comp0 (a string); > insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm"); > insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm"); > create external table comp1 stored as orc as select * from comp0; > insert into comp0 values ("qwertyuiopasdfghjklzxcvbnm"); > create external table comp2 stored as orc as select * from comp0; > create table comp3 (a string); > load data inpath '/warehouse/tablespace/external/hive/comp1' into table comp3; > load data inpath '/warehouse/tablespace/external/hive/comp2' into table > comp3;{code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008729#comment-17008729 ] Laszlo Pinter commented on HIVE-22535: -- Meanwhile it turned out, that CompactionTxnHandler#findNextToCompact() already does this check, therefore I'm closing this jira. > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch, HIVE-22535.02.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Resolution: Won't Fix Status: Resolved (was: Patch Available) > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch, HIVE-22535.02.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.04.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch, HIVE-20934.04.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.12.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch, HIVE-22280.10.patch, HIVE-22280.11.patch, > HIVE-22280.12.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Attachment: HIVE-22535.02.patch > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch, HIVE-22535.02.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.03.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch, > HIVE-20934.03.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.11.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch, HIVE-22280.10.patch, HIVE-22280.11.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Attachment: HIVE-22535.01.patch > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22535) ACID: Check for already running compactions in Worker
[ https://issues.apache.org/jira/browse/HIVE-22535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22535: - Status: Patch Available (was: Open) > ACID: Check for already running compactions in Worker > -- > > Key: HIVE-22535 > URL: https://issues.apache.org/jira/browse/HIVE-22535 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22535.01.patch > > > Compactor worker doesn't check if a compaction is already running for a given > resource. Initiator does this, but in case of manual compaction, this check > is missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21603) Java 11 preparation: update powermock version
[ https://issues.apache.org/jira/browse/HIVE-21603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-21603: Assignee: (was: Laszlo Pinter) > Java 11 preparation: update powermock version > - > > Key: HIVE-21603 > URL: https://issues.apache.org/jira/browse/HIVE-21603 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Priority: Major > > PowerMock1 has no support for Java11, therefore we need to bump its version > to 2.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.02.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch, HIVE-20934.02.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.10.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch, HIVE-22280.10.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22582) Avoid reading table as ACID when table name is starting with "delta" , but table is not transactional and BI Split Strategy is used
[ https://issues.apache.org/jira/browse/HIVE-22582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988684#comment-16988684 ] Laszlo Pinter commented on HIVE-22582: -- +1 (non-binding) > Avoid reading table as ACID when table name is starting with "delta" , but > table is not transactional and BI Split Strategy is used > --- > > Key: HIVE-22582 > URL: https://issues.apache.org/jira/browse/HIVE-22582 > Project: Hive > Issue Type: Bug >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-22582.patch > > > The issue is fixed in HIVE-22473 but missed a check for BI Split Strategy. > Steps to reproduce: > {code:java} > set hive.exec.orc.split.strategy=BI; > create table delta_result (a int) stored as orc > tblproperties('transactional'='false'); > insert into delta_result select 1; > select * from delta_result; > {code} > Exception Stack Trace: > {code:java} > Caused by: java.lang.RuntimeException: ORC split generation failed with > exception: String index out of range: -1 > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:461) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:430) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:336) > at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:576) > ... 50 more > Caused by: java.lang.StringIndexOutOfBoundsException: String index out of > range: -1 > at java.lang.String.substring(String.java:1967) > at > org.apache.hadoop.hive.ql.io.AcidUtils.parsedDelta(AcidUtils.java:1128) > at > org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight.parse(AcidUtils.java:921) > at > org.apache.hadoop.hive.ql.io.AcidUtils.getLogicalLength(AcidUtils.java:2084) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$BISplitStrategy.getSplits(OrcInputFormat.java:1115) > at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1905) > ... 55 more > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22473) Avoid reading table as ACID when table name is starting with "delta", but table is not transactional
[ https://issues.apache.org/jira/browse/HIVE-22473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988597#comment-16988597 ] Laszlo Pinter commented on HIVE-22473: -- [~aditya-shah] Please open a new jira, and describe the details. Thanks > Avoid reading table as ACID when table name is starting with "delta", but > table is not transactional > > > Key: HIVE-22473 > URL: https://issues.apache.org/jira/browse/HIVE-22473 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22473.01.patch, HIVE-22473.02.patch, > HIVE-22473.03.patch, HIVE-22473.04.patch > > > {code:sql} > create table delta_result (a int) stored as orc > tblproperties('transactional'='false'); > insert into delta_result select 1; > select * from delta_result; > {code} > The above query will result in the following exception: > 2019-11-08T13:49:05,780 WARN [HiveServer2-Handler-Pool: Thread-7906] > thrift.ThriftCLIService: Error fetching results:2019-11-08T13:49:05,780 WARN > [HiveServer2-Handler-Pool: Thread-7906] thrift.ThriftCLIService: Error > fetching results:org.apache.hive.service.cli.HiveSQLException: > java.io.IOException: java.lang.RuntimeException: ORC split generation failed > with exception: java.lang.StringIndexOutOfBoundsException: String index out > of range: -1 at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:481) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.cli.operation.OperationManager.getOperationNextRowSet(OperationManager.java:331) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:946) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.cli.CLIService.fetchResults(CLIService.java:567) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.cli.thrift.ThriftCLIService.FetchResults(ThriftCLIService.java:801) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1837) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults.getResult(TCLIService.java:1822) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > ~[?:1.8.0_211] at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]Caused > by: java.io.IOException: java.lang.RuntimeException: ORC split generation > failed with exception: java.lang.StringIndexOutOfBoundsException: String > index out of range: -1 at > org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:638) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:545) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:151) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2142) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:241) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:476) > ~[hive-service-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] ... 13 moreCaused by: > java.lang.RuntimeException: ORC split generation failed with exception: > java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1929) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:2016) > ~[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT] at > org.
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.07.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch, HIVE-22534.07.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.09.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch, > HIVE-22280.09.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.06.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch, > HIVE-22534.06.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986845#comment-16986845 ] Laszlo Pinter commented on HIVE-22554: -- Thanks [~pvary], for the review & commit! > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22554.01.patch, HIVE-22554.02.patch > > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Status: Patch Available (was: Open) > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20934) ACID: Query based compactor for minor compaction
[ https://issues.apache.org/jira/browse/HIVE-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-20934: - Attachment: HIVE-20934.01.patch > ACID: Query based compactor for minor compaction > > > Key: HIVE-20934 > URL: https://issues.apache.org/jira/browse/HIVE-20934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20934.01.patch > > > Follow up of HIVE-20699. This is to enable running minor compactions as a > HiveQL query -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Status: Patch Available (was: Reopened) > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22280: - Attachment: HIVE-22280.08.patch > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch, HIVE-22280.08.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reopened HIVE-22280: -- > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22554: - Attachment: HIVE-22554.02.patch > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22554.01.patch, HIVE-22554.02.patch > > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22554: - Attachment: HIVE-22554.01.patch > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22554.01.patch > > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22554: - Attachment: (was: HIVE-22554.01.patch) > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22554.01.patch > > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22554: - Attachment: HIVE-22554.01.patch > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22554.01.patch > > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22554: - Status: Patch Available (was: Open) > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22554.01.patch > > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-21765) Support partitions on temporary tables
[ https://issues.apache.org/jira/browse/HIVE-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter resolved HIVE-21765. -- Resolution: Fixed > Support partitions on temporary tables > -- > > Key: HIVE-21765 > URL: https://issues.apache.org/jira/browse/HIVE-21765 > Project: Hive > Issue Type: New Feature >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > > HIVE-18739 and HIVE-20661 added partial support for partition columns on > temporary tables, but it was not complete and it was available only for > internal usage. The scope of this jira, is to implement the missing parts and > expose this functionality. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22280) Q tests for partitioned temporary tables
[ https://issues.apache.org/jira/browse/HIVE-22280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984403#comment-16984403 ] Laszlo Pinter commented on HIVE-22280: -- [~pvary] Thanks for the review and commit. > Q tests for partitioned temporary tables > > > Key: HIVE-22280 > URL: https://issues.apache.org/jira/browse/HIVE-22280 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22280.01.patch, HIVE-22280.02.patch, > HIVE-22280.03.patch, HIVE-22280.04.patch, HIVE-22280.05.patch, > HIVE-22280.06.patch, HIVE-22280.07.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-22542) Update orc version to 1.5.8
[ https://issues.apache.org/jira/browse/HIVE-22542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter resolved HIVE-22542. -- Resolution: Duplicate > Update orc version to 1.5.8 > --- > > Key: HIVE-22542 > URL: https://issues.apache.org/jira/browse/HIVE-22542 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22542.01.patch > > > Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22542) Update orc version to 1.5.8
[ https://issues.apache.org/jira/browse/HIVE-22542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22542: - Status: Open (was: Patch Available) > Update orc version to 1.5.8 > --- > > Key: HIVE-22542 > URL: https://issues.apache.org/jira/browse/HIVE-22542 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22542.01.patch > > > Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22554) ACID: Wait timeout for blocking compaction should be configurable
[ https://issues.apache.org/jira/browse/HIVE-22554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter reassigned HIVE-22554: > ACID: Wait timeout for blocking compaction should be configurable > - > > Key: HIVE-22554 > URL: https://issues.apache.org/jira/browse/HIVE-22554 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > > The wait timeout for blocking compaction is hardcoded to 5 minutes. > {code:java} > public class AlterTableCompactOperation extends > DDLOperation { > private static final int FIVE_MINUTES_IN_MILLIES = 5*60*1000; > ... > }{code} > This should be configurable via a Hive Configuration parameter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.05.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch, HIVE-22534.05.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22534) ACID: Improve Compactor thread logging
[ https://issues.apache.org/jira/browse/HIVE-22534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22534: - Attachment: HIVE-22534.04.patch > ACID: Improve Compactor thread logging > -- > > Key: HIVE-22534 > URL: https://issues.apache.org/jira/browse/HIVE-22534 > Project: Hive > Issue Type: Improvement >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Minor > Attachments: HIVE-22534.01.patch, HIVE-22534.02.patch, > HIVE-22534.03.patch, HIVE-22534.04.patch > > > Make sure that it is easy to find issues when one of the compactor thread > fails. > Maybe: > * MDC - with iteration / threadname - so we can easily grep the logs for a > given run > * MDC with table/partition data on which the worker is working -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22542) Update orc version to 1.5.8
[ https://issues.apache.org/jira/browse/HIVE-22542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-22542: - Attachment: HIVE-22542.01.patch > Update orc version to 1.5.8 > --- > > Key: HIVE-22542 > URL: https://issues.apache.org/jira/browse/HIVE-22542 > Project: Hive > Issue Type: Task >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-22542.01.patch > > > Update orc to 1.5.8 in root pom -- This message was sent by Atlassian Jira (v8.3.4#803005)