[jira] [Work logged] (HIVE-26329) Upgrade protobuf-java to 3.16.1

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26329?focusedWorklogId=804088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804088
 ]

ASF GitHub Bot logged work on HIVE-26329:
-

Author: ASF GitHub Bot
Created on: 27/Aug/22 00:21
Start Date: 27/Aug/22 00:21
Worklog Time Spent: 10m 
  Work Description: github-actions[bot] closed pull request #3377: 
HIVE-26329: Upgrade protobuf-java to 3.16.1
URL: https://github.com/apache/hive/pull/3377




Issue Time Tracking
---

Worklog Id: (was: 804088)
Time Spent: 20m  (was: 10m)

> Upgrade protobuf-java to 3.16.1
> ---
>
> Key: HIVE-26329
> URL: https://issues.apache.org/jira/browse/HIVE-26329
> Project: Hive
>  Issue Type: Improvement
>Reporter: Hongdan Zhu
>Assignee: Hongdan Zhu
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (HIVE-26495) MSCK repair perf issue HMSChecker ThreadPool is blocked at fs.listStatus

2022-08-26 Thread Naresh P R (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-26495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585622#comment-17585622
 ] 

Naresh P R commented on HIVE-26495:
---

[~srahman] I dint see much improvement. I was missing HIVE-21040 which helped 
to improve performance in S3 table with millions of files under partition path.

But still i wanted to pursue with the patch. Let me know if you can review it.

> MSCK repair perf issue HMSChecker ThreadPool is blocked at fs.listStatus
> 
>
> Key: HIVE-26495
> URL: https://issues.apache.org/jira/browse/HIVE-26495
> Project: Hive
>  Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> With hive.metastore.fshandler.threads = 15, all 15 *MSCK-GetPaths-xx* are 
> slogging at following trace.
> {code:java}
> "MSCK-GetPaths-11" #12345 daemon prio=5 os_prio=0 tid= nid= waiting on 
> condition [0x7f9f099a6000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x0003f92d1668> (a 
> java.util.concurrent.CompletableFuture$Signaller)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>     at 
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
>     at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> ...
> at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3230)
>     at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
>     at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1995)
>     at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.processPathDepthInfo(HiveMetaStoreChecker.java:550)
>     at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:543)
>     at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:525)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:750){code}
> We should take advantage of non-block listStatusIterator instead of 
> listStatus which is a blocking call.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (HIVE-26495) MSCK repair perf issue HMSChecker ThreadPool is blocked at fs.listStatus

2022-08-26 Thread Naresh P R (Jira) 


[ 
https://issues.apache.org/jira/browse/HIVE-26495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585622#comment-17585622
 ] 

Naresh P R edited comment on HIVE-26495 at 8/26/22 9:52 PM:


[~srahman] I dint see much improvement. I was missing HIVE-21040 in my internal 
branch which helped to improve performance in S3 table with millions of files 
under partition path.

But still i wanted to pursue with the patch. Let me know if you can review it.


was (Author: nareshpr):
[~srahman] I dint see much improvement. I was missing HIVE-21040 which helped 
to improve performance in S3 table with millions of files under partition path.

But still i wanted to pursue with the patch. Let me know if you can review it.

> MSCK repair perf issue HMSChecker ThreadPool is blocked at fs.listStatus
> 
>
> Key: HIVE-26495
> URL: https://issues.apache.org/jira/browse/HIVE-26495
> Project: Hive
>  Issue Type: Bug
>Reporter: Naresh P R
>Assignee: Naresh P R
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> With hive.metastore.fshandler.threads = 15, all 15 *MSCK-GetPaths-xx* are 
> slogging at following trace.
> {code:java}
> "MSCK-GetPaths-11" #12345 daemon prio=5 os_prio=0 tid= nid= waiting on 
> condition [0x7f9f099a6000]
>    java.lang.Thread.State: WAITING (parking)
>     at sun.misc.Unsafe.park(Native Method)
>     - parking to wait for  <0x0003f92d1668> (a 
> java.util.concurrent.CompletableFuture$Signaller)
>     at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>     at 
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
>     at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> ...
> at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:3230)
>     at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1953)
>     at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1995)
>     at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.processPathDepthInfo(HiveMetaStoreChecker.java:550)
>     at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:543)
>     at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreChecker$PathDepthInfoCallable.call(HiveMetaStoreChecker.java:525)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:750){code}
> We should take advantage of non-block listStatusIterator instead of 
> listStatus which is a blocking call.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26312) Use default digest normalization strategy in CBO

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26312?focusedWorklogId=804056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804056
 ]

ASF GitHub Bot logged work on HIVE-26312:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 18:50
Start Date: 26/Aug/22 18:50
Worklog Time Spent: 10m 
  Work Description: soumyakanti3578 commented on code in PR #3359:
URL: https://github.com/apache/hive/pull/3359#discussion_r956341300


##
ql/src/test/results/clientpositive/llap/vector_interval_2.q.out:
##
@@ -1350,16 +1350,16 @@ STAGE PLANS:
 Map Operator Tree:
 TableScan
   alias: vector_interval_2
-  filterExpr: (CAST( str1 AS INTERVAL YEAR TO MONTH) is not 
null and ((dt + CAST( str1 AS INTERVAL YEAR TO MONTH)) = DATE'2002-03-01') and 
((dt + INTERVAL'1-2') = DATE'2002-03-01') and (dt <> (dt + CAST( str1 AS 
INTERVAL YEAR TO MONTH))) and (dt <> (dt + INTERVAL'1-2'))) (type: boolean)
+  filterExpr: (((dt + CAST( str1 AS INTERVAL YEAR TO MONTH)) = 
DATE'2002-03-01') and ((dt + INTERVAL'1-2') = DATE'2002-03-01') and (dt <> (dt 
+ CAST( str1 AS INTERVAL YEAR TO MONTH))) and (dt <> (dt + INTERVAL'1-2'))) 
(type: boolean)

Review Comment:
   I think I figured out why `is not null` is missing here. I tried to 
reproduce this with the following test:
   
   > explain vectorization expression
   select ts from vector_interval_2
   where
 timestamp '2001-01-01 01:02:03' <= dt + interval '0 1:2:3' day to second
 and dt + interval '0 1:2:3' day to second = timestamp '2001-01-01 01:02:03'
 and dt + interval '0 1:2:3' day to second >= timestamp '2001-01-01 
01:02:03'
   order by ts;
   
   With default value for `CalciteSystemProperty.ENABLE_REX_DIGEST_NORMALIZE`, 
we can see 
[here](https://github.com/apache/calcite/blob/calcite-1.25.0/core/src/main/java/org/apache/calcite/rex/RexNormalize.java#L123)
 that `<, <=` are preferred over `>, >=`, i.e., if possible all `>, >=` are 
reversed. Now, the `AND` clause contains expressions like `(AA) and (D>=C)`, for example, just above this diff, there's the diff on line 
1172 which essentially has:
   >str3 <= str4 and str3 < str4 and str4 >= str3 and str4 > str3
   
   and we can see that the second half is pruned as it is exactly the same as 
the first half. These duplicate expressions are not getting added in the 
`digests` 
[here](https://github.com/apache/calcite/blob/calcite-1.25.0/core/src/main/java/org/apache/calcite/rex/RexUtil.java#L1142).
 `digests.add` eventually leads to  `RexNormalize.hashcode` method, which 
reverses the `>, >=` rexnodes and returns same hashcode as the `<, <=` 
expressions and thus they are not added to `digests`. 
   
   Now, when `CalciteSystemProperty.ENABLE_REX_DIGEST_NORMALIZE` is set to 
`false`, we get all these duplicate expressions which are not pruned, and in 
`RexSimplify.simplifyUsingPredicates` method, we calculate `range2` 
[here](https://github.com/apache/calcite/blob/calcite-1.25.0/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1631),
 by calling `residue` and passing the list of predicates 
`predicates.pulledUpPredicates`. This list contains all the predicates that 
have been seen before the current predicate. So when there are duplicates and 
we see the same predicate for the second time, `residue` returns `Range.all()` 
from 
[here](https://github.com/apache/calcite/blob/calcite-1.25.0/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1695),
 and this results in a `is not null` 
[here](https://github.com/apache/calcite/blob/calcite-1.25.0/core/src/main/java/org/apache/calcite/rex/RexSimplify.java#L1643).
   
   I think all the changes in this commit are fine as all of them are because 
of pruning of duplicate predicates. What do you think @zabetak ?





Issue Time Tracking
---

Worklog Id: (was: 804056)
Time Spent: 40m  (was: 0.5h)

> Use default digest normalization strategy in CBO
> 
>
> Key: HIVE-26312
> URL: https://issues.apache.org/jira/browse/HIVE-26312
> Project: Hive
>  Issue Type: Task
>  Components: CBO
>Affects Versions: 4.0.0-alpha-1
>Reporter: Stamatis Zampetakis
>Assignee: Stamatis Zampetakis
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> CALCITE-2450 introduced a way to improve planning time by normalizing some 
> query expressions (RexNodes). The behavior can be enabled/disabled via the 
> following system property: calcite.enable.rexnode.digest.normalize
> There was an attempt to disable the normalization explicitly in HIVE-23456 to 
> avoid rendering HiveFilterSortPredicates rule useless. However, the [way the 
> normalization is

[jira] [Work logged] (HIVE-26474) Enable converting insert-only tables to full ACID tables

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26474?focusedWorklogId=804034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804034
 ]

ASF GitHub Bot logged work on HIVE-26474:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 17:51
Start Date: 26/Aug/22 17:51
Worklog Time Spent: 10m 
  Work Description: sonarcloud[bot] commented on PR #3550:
URL: https://github.com/apache/hive/pull/3550#issuecomment-1228765312

   Kudos, SonarCloud Quality Gate passed!  [![Quality Gate 
passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png
 'Quality Gate 
passed')](https://sonarcloud.io/dashboard?id=apache_hive=3550)
   
   
[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png
 
'Bug')](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=BUG)
 
[![C](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/C-16px.png
 
'C')](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=BUG)
 [1 
Bug](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=BUG)
  
   
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png
 
'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=VULNERABILITY)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=VULNERABILITY)
 [0 
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=VULNERABILITY)
  
   [![Security 
Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png
 'Security 
Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3550=false=SECURITY_HOTSPOT)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3550=false=SECURITY_HOTSPOT)
 [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3550=false=SECURITY_HOTSPOT)
  
   [![Code 
Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png
 'Code 
Smell')](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=CODE_SMELL)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=CODE_SMELL)
 [5 Code 
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3550=false=CODE_SMELL)
   
   [![No Coverage 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png
 'No Coverage 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3550=coverage=list)
 No Coverage information  
   [![No Duplication 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png
 'No Duplication 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3550=duplicated_lines_density=list)
 No Duplication information
   
   




Issue Time Tracking
---

Worklog Id: (was: 804034)
Time Spent: 2.5h  (was: 2h 20m)

> Enable converting insert-only tables to full ACID tables
> 
>
> Key: HIVE-26474
> URL: https://issues.apache.org/jira/browse/HIVE-26474
> Project: Hive
>  Issue Type: Improvement
>Reporter: Simhadri Govindappa
>Assignee: Simhadri Govindappa
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2.5h
>  Remaining Estimate: 0h
>
> Currently we can convert a non-ACID table to full ACID via altering 
> TBLPROPERTIES ("transactional"="false" -> "transactional"="true"), but we 
> don't support converting insert-only tables to full ACID tables 
> ("transactional_properties"="insert_only" - > 
> "transactional_properties"="default").



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=804025=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-804025
 ]

ASF GitHub Bot logged work on HIVE-26498:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 17:31
Start Date: 26/Aug/22 17:31
Worklog Time Spent: 10m 
  Work Description: sonarcloud[bot] commented on PR #3552:
URL: https://github.com/apache/hive/pull/3552#issuecomment-1228749668

   Kudos, SonarCloud Quality Gate passed!  [![Quality Gate 
passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png
 'Quality Gate 
passed')](https://sonarcloud.io/dashboard?id=apache_hive=3552)
   
   
[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png
 
'Bug')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
 [0 
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
  
   
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png
 
'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
 [0 
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
  
   [![Security 
Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png
 'Security 
Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
 [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
  
   [![Code 
Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png
 'Code 
Smell')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
 [14 Code 
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
   
   [![No Coverage 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png
 'No Coverage 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3552=coverage=list)
 No Coverage information  
   [![No Duplication 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png
 'No Duplication 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3552=duplicated_lines_density=list)
 No Duplication information
   
   




Issue Time Tracking
---

Worklog Id: (was: 804025)
Time Spent: 50m  (was: 40m)

> Implement MV maintenance with Iceberg sources using full rebuild
> 
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg 
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52), 
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
>   HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26481) Cleaner fails with FileNotFoundException

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26481?focusedWorklogId=803977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803977
 ]

ASF GitHub Bot logged work on HIVE-26481:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 15:02
Start Date: 26/Aug/22 15:02
Worklog Time Spent: 10m 
  Work Description: sonarcloud[bot] commented on PR #3531:
URL: https://github.com/apache/hive/pull/3531#issuecomment-1228606902

   Kudos, SonarCloud Quality Gate passed!  [![Quality Gate 
passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png
 'Quality Gate 
passed')](https://sonarcloud.io/dashboard?id=apache_hive=3531)
   
   
[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png
 
'Bug')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=BUG)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=BUG)
 [0 
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=BUG)
  
   
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png
 
'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=VULNERABILITY)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=VULNERABILITY)
 [0 
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=VULNERABILITY)
  
   [![Security 
Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png
 'Security 
Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3531=false=SECURITY_HOTSPOT)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3531=false=SECURITY_HOTSPOT)
 [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3531=false=SECURITY_HOTSPOT)
  
   [![Code 
Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png
 'Code 
Smell')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=CODE_SMELL)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=CODE_SMELL)
 [0 Code 
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=CODE_SMELL)
   
   [![No Coverage 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png
 'No Coverage 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3531=coverage=list)
 No Coverage information  
   [![No Duplication 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png
 'No Duplication 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3531=duplicated_lines_density=list)
 No Duplication information
   
   




Issue Time Tracking
---

Worklog Id: (was: 803977)
Time Spent: 3.5h  (was: 3h 20m)

> Cleaner fails with FileNotFoundException
> 
>
> Key: HIVE-26481
> URL: https://issues.apache.org/jira/browse/HIVE-26481
> Project: Hive
>  Issue Type: Bug
>Reporter: KIRTI RUGE
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> The compaction fails when the Cleaner tried to remove a missing directory 
> from HDFS.
> {code:java}
> 2022-08-05 18:56:38,873 INFO org.apache.hadoop.hive.ql.txn.compactor.Cleaner: 
> [Cleaner-executor-thread-0]: Starting cleaning for 
> id:30,dbname:default,tableName:test_concur_compaction_minor,partName:null,state:�,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:4,errorMessage:null,workerId:
>  null,initiatorId: null 2022-08-05 18:56:38,888 ERROR 
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner: [Cleaner-executor-thread-0]: 
> Caught exception when cleaning, unable to complete cleaning of 
> id:30,dbname:default,tableName:test_concur_compaction_minor,partName:null,state:�,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:4,errorMessage:null,workerId:
>  null,initiatorId: null java.io.FileNotFoundException: File 
>

[jira] [Work logged] (HIVE-26046) MySQL's bit datatype is default to void datatype in hive

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26046?focusedWorklogId=803973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803973
 ]

ASF GitHub Bot logged work on HIVE-26046:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 14:48
Start Date: 26/Aug/22 14:48
Worklog Time Spent: 10m 
  Work Description: nrg4878 commented on code in PR #3276:
URL: https://github.com/apache/hive/pull/3276#discussion_r956125849


##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/MySQLConnectorProvider.java:
##
@@ -90,10 +90,20 @@ protected String getDataType(String dbDataType, int size) {
 // map any db specific types here.
 switch (dbDataType.toLowerCase())
 {
+case "bit":
+  return toHiveBitType(size);
 default:
   mappedType = ColumnType.VOID_TYPE_NAME;
   break;
 }
 return mappedType;
   }
+
+  private String toHiveBitType(int size) {
+if (size <= 1) {
+  return ColumnType.BOOLEAN_TYPE_NAME;
+} else {
+  return ColumnType.BIGINT_TYPE_NAME;

Review Comment:
   @zhangbutao is option #1 above not possible where we set the hive.sql.query 
to include like a select bin(column_name) ... on the table containing bit type 
columns?
   
   mapping to hive binary type might work. One concern is that doesn't hive 
expect binary values to be within ''. So something like '0111' instead of 0111. 
Can you please try it out to see how this works end to end? 





Issue Time Tracking
---

Worklog Id: (was: 803973)
Time Spent: 3h 10m  (was: 3h)

> MySQL's bit datatype is default to void datatype in hive
> 
>
> Key: HIVE-26046
> URL: https://issues.apache.org/jira/browse/HIVE-26046
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> describe on a table that contains a "bit" datatype gets mapped to void. We 
> need a explicit conversion logic in the MySQL ConnectorProvider to map it to 
> a suitable datatype in hive.
> {noformat}
> +---+---++
> |   col_name| data_type   
>   |  comment   |
> +---+---++
> | tbl_id| bigint  
>   | from deserializer  |
> | create_time   | int 
>   | from deserializer  |
> | db_id | bigint  
>   | from deserializer  |
> | last_access_time  | int 
>   | from deserializer  |
> | owner | varchar(767)
>   | from deserializer  |
> | owner_type| varchar(10) 
>   | from deserializer  |
> | retention | int 
>   | from deserializer  |
> | sd_id | bigint  
>   | from deserializer  |
> | tbl_name  | varchar(256)
>   | from deserializer  |
> | tbl_type  | varchar(128)
>   | from deserializer  |
> | view_expanded_text| string  
>   | from deserializer  |
> | view_original_text| string  
>   | from deserializer  |
> | is_rewrite_enabled| void
>   | from deserializer  |
> | write_id  | bigint  
>   | from deserializer  
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26498) Implement MV maintenance with Iceberg sources using full rebuild

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26498?focusedWorklogId=803949=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803949
 ]

ASF GitHub Bot logged work on HIVE-26498:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 12:37
Start Date: 26/Aug/22 12:37
Worklog Time Spent: 10m 
  Work Description: sonarcloud[bot] commented on PR #3552:
URL: https://github.com/apache/hive/pull/3552#issuecomment-1228436428

   Kudos, SonarCloud Quality Gate passed!  [![Quality Gate 
passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png
 'Quality Gate 
passed')](https://sonarcloud.io/dashboard?id=apache_hive=3552)
   
   
[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png
 
'Bug')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
 [0 
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=BUG)
  
   
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png
 
'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
 [0 
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=VULNERABILITY)
  
   [![Security 
Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png
 'Security 
Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
 [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3552=false=SECURITY_HOTSPOT)
  
   [![Code 
Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png
 'Code 
Smell')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
 [16 Code 
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3552=false=CODE_SMELL)
   
   [![No Coverage 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png
 'No Coverage 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3552=coverage=list)
 No Coverage information  
   [![No Duplication 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png
 'No Duplication 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3552=duplicated_lines_density=list)
 No Duplication information
   
   




Issue Time Tracking
---

Worklog Id: (was: 803949)
Time Spent: 40m  (was: 0.5h)

> Implement MV maintenance with Iceberg sources using full rebuild
> 
>
> Key: HIVE-26498
> URL: https://issues.apache.org/jira/browse/HIVE-26498
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create external table tbl_ice(a int, b string, c int) stored by iceberg 
> stored as orc tblproperties ('format-version'='2');
> insert into tbl_ice values (1, 'one', 50), (2, 'two', 51), (3, 'three', 52), 
> (4, 'four', 53), (5, 'five', 54);
> create materialized view mat1 as
> select b, c from tbl_ice where c > 52;
> insert into tbl_ice values (111, 'one', 55), (333, 'two', 56);
> explain cbo
> alter materialized view mat1 rebuild;
> alter materialized view mat1 rebuild;
> {code}
> MV full rebuild plan
> {code}
> CBO PLAN:
> HiveProject(b=[$1], c=[$2])
>   HiveFilter(condition=[>($2, 52)])
> HiveTableScan(table=[[default, tbl_ice]], table:alias=[tbl_ice])
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26494) Fix flaky test TestJdbcWithMiniHS2 testHttpRetryOnServerIdleTimeout

2022-08-26 Thread Zhihua Deng (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zhihua Deng resolved HIVE-26494.

Fix Version/s: 4.0.0
   4.0.0-alpha-2
 Assignee: Zhihua Deng
   Resolution: Fixed

Merged into master. Thank you [~ayushtkn]  for the review!

> Fix flaky test TestJdbcWithMiniHS2 testHttpRetryOnServerIdleTimeout
> ---
>
> Key: HIVE-26494
> URL: https://issues.apache.org/jira/browse/HIVE-26494
> Project: Hive
>  Issue Type: Test
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-2
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The TestJdbcWithMiniHS2#testHttpRetryOnServerIdleTimeout fails on master:
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/1362/tests]
> It can be fixed by setting hive.server2.thrift.http.max.idle.time to a larger 
> value, other than 5ms.
> Flaky check: http://ci.hive.apache.org/job/hive-flaky-check/585/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26494) Fix flaky test TestJdbcWithMiniHS2 testHttpRetryOnServerIdleTimeout

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26494?focusedWorklogId=803944=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803944
 ]

ASF GitHub Bot logged work on HIVE-26494:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 12:19
Start Date: 26/Aug/22 12:19
Worklog Time Spent: 10m 
  Work Description: dengzhhu653 merged PR #3547:
URL: https://github.com/apache/hive/pull/3547




Issue Time Tracking
---

Worklog Id: (was: 803944)
Time Spent: 40m  (was: 0.5h)

> Fix flaky test TestJdbcWithMiniHS2 testHttpRetryOnServerIdleTimeout
> ---
>
> Key: HIVE-26494
> URL: https://issues.apache.org/jira/browse/HIVE-26494
> Project: Hive
>  Issue Type: Test
>Reporter: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The TestJdbcWithMiniHS2#testHttpRetryOnServerIdleTimeout fails on master:
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/1362/tests]
> It can be fixed by setting hive.server2.thrift.http.max.idle.time to a larger 
> value, other than 5ms.
> Flaky check: http://ci.hive.apache.org/job/hive-flaky-check/585/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-24933) Replication fails for transactional tables having same name as dropped non-transactional table

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-24933?focusedWorklogId=803936=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803936
 ]

ASF GitHub Bot logged work on HIVE-24933:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 11:59
Start Date: 26/Aug/22 11:59
Worklog Time Spent: 10m 
  Work Description: sonarcloud[bot] commented on PR #3544:
URL: https://github.com/apache/hive/pull/3544#issuecomment-1228402961

   Kudos, SonarCloud Quality Gate passed!  [![Quality Gate 
passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png
 'Quality Gate 
passed')](https://sonarcloud.io/dashboard?id=apache_hive=3544)
   
   
[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png
 
'Bug')](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=BUG)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=BUG)
 [0 
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=BUG)
  
   
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png
 
'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=VULNERABILITY)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=VULNERABILITY)
 [0 
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=VULNERABILITY)
  
   [![Security 
Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png
 'Security 
Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3544=false=SECURITY_HOTSPOT)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3544=false=SECURITY_HOTSPOT)
 [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3544=false=SECURITY_HOTSPOT)
  
   [![Code 
Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png
 'Code 
Smell')](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=CODE_SMELL)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=CODE_SMELL)
 [35 Code 
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3544=false=CODE_SMELL)
   
   [![No Coverage 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png
 'No Coverage 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3544=coverage=list)
 No Coverage information  
   [![No Duplication 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png
 'No Duplication 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3544=duplicated_lines_density=list)
 No Duplication information
   
   




Issue Time Tracking
---

Worklog Id: (was: 803936)
Time Spent: 5h 10m  (was: 5h)

> Replication fails for transactional tables having same name as dropped 
> non-transactional table
> --
>
> Key: HIVE-24933
> URL: https://issues.apache.org/jira/browse/HIVE-24933
> Project: Hive
>  Issue Type: Bug
>Reporter: Pratyush Madhukar
>Assignee: Pratyush Madhukar
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26468) Cleanup dependency to FlatBuffers

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26468?focusedWorklogId=803914=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803914
 ]

ASF GitHub Bot logged work on HIVE-26468:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 11:14
Start Date: 26/Aug/22 11:14
Worklog Time Spent: 10m 
  Work Description: slachiewicz commented on code in PR #3517:
URL: https://github.com/apache/hive/pull/3517#discussion_r955937007


##
serde/pom.xml:
##
@@ -81,7 +81,7 @@
   ${hppc.version}
 
 
-  com.github.davidmoten
+  com.google.flatbuffers

Review Comment:
   I don't think so (see warning message in jira ticket), previously for this 
module we have 2 dependencies from different groups and shade plugin may select 
one of them randomly. My change makes clear to use only official artefacts from 
one source.





Issue Time Tracking
---

Worklog Id: (was: 803914)
Time Spent: 0.5h  (was: 20m)

> Cleanup dependency to FlatBuffers
> -
>
> Key: HIVE-26468
> URL: https://issues.apache.org/jira/browse/HIVE-26468
> Project: Hive
>  Issue Type: Bug
>Reporter: Sylwester Lachiewicz
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Remove issue with overlapping dependencies to FlatBuffers. 
> Warning inside shading step for hive-jdbc, hive-beeline shows that we have 
> wring dependencies. Similar issue exists inside hive-serde.
>  
> Simply switch to Google FlatBuffers artefact
>  
> [WARNING] flatbuffers-java-1.9.0.jar, flatbuffers-java-1.6.0.1.jar define 5 
> overlapping classes:
> [WARNING]   - com.google.flatbuffers.Struct
> [WARNING]   - com.google.flatbuffers.Table$1
> [WARNING]   - com.google.flatbuffers.Constants
> [WARNING]   - com.google.flatbuffers.Table
> [WARNING]   - com.google.flatbuffers.FlatBufferBuilder



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26468) Cleanup dependency to FlatBuffers

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26468?focusedWorklogId=803913=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803913
 ]

ASF GitHub Bot logged work on HIVE-26468:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 11:10
Start Date: 26/Aug/22 11:10
Worklog Time Spent: 10m 
  Work Description: ayushtkn commented on code in PR #3517:
URL: https://github.com/apache/hive/pull/3517#discussion_r955933414


##
serde/pom.xml:
##
@@ -81,7 +81,7 @@
   ${hppc.version}
 
 
-  com.github.davidmoten
+  com.google.flatbuffers

Review Comment:
   does this change introduce any incompatibilities?





Issue Time Tracking
---

Worklog Id: (was: 803913)
Time Spent: 20m  (was: 10m)

> Cleanup dependency to FlatBuffers
> -
>
> Key: HIVE-26468
> URL: https://issues.apache.org/jira/browse/HIVE-26468
> Project: Hive
>  Issue Type: Bug
>Reporter: Sylwester Lachiewicz
>Priority: Minor
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Remove issue with overlapping dependencies to FlatBuffers. 
> Warning inside shading step for hive-jdbc, hive-beeline shows that we have 
> wring dependencies. Similar issue exists inside hive-serde.
>  
> Simply switch to Google FlatBuffers artefact
>  
> [WARNING] flatbuffers-java-1.9.0.jar, flatbuffers-java-1.6.0.1.jar define 5 
> overlapping classes:
> [WARNING]   - com.google.flatbuffers.Struct
> [WARNING]   - com.google.flatbuffers.Table$1
> [WARNING]   - com.google.flatbuffers.Constants
> [WARNING]   - com.google.flatbuffers.Table
> [WARNING]   - com.google.flatbuffers.FlatBufferBuilder



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26481) Cleaner fails with FileNotFoundException

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26481?focusedWorklogId=803885=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803885
 ]

ASF GitHub Bot logged work on HIVE-26481:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 09:32
Start Date: 26/Aug/22 09:32
Worklog Time Spent: 10m 
  Work Description: sonarcloud[bot] commented on PR #3531:
URL: https://github.com/apache/hive/pull/3531#issuecomment-1228278240

   Kudos, SonarCloud Quality Gate passed!  [![Quality Gate 
passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png
 'Quality Gate 
passed')](https://sonarcloud.io/dashboard?id=apache_hive=3531)
   
   
[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png
 
'Bug')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=BUG)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=BUG)
 [0 
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=BUG)
  
   
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png
 
'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=VULNERABILITY)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=VULNERABILITY)
 [0 
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=VULNERABILITY)
  
   [![Security 
Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png
 'Security 
Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3531=false=SECURITY_HOTSPOT)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3531=false=SECURITY_HOTSPOT)
 [0 Security 
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3531=false=SECURITY_HOTSPOT)
  
   [![Code 
Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png
 'Code 
Smell')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=CODE_SMELL)
 
[![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png
 
'A')](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=CODE_SMELL)
 [0 Code 
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3531=false=CODE_SMELL)
   
   [![No Coverage 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png
 'No Coverage 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3531=coverage=list)
 No Coverage information  
   [![No Duplication 
information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png
 'No Duplication 
information')](https://sonarcloud.io/component_measures?id=apache_hive=3531=duplicated_lines_density=list)
 No Duplication information
   
   




Issue Time Tracking
---

Worklog Id: (was: 803885)
Time Spent: 3h 20m  (was: 3h 10m)

> Cleaner fails with FileNotFoundException
> 
>
> Key: HIVE-26481
> URL: https://issues.apache.org/jira/browse/HIVE-26481
> Project: Hive
>  Issue Type: Bug
>Reporter: KIRTI RUGE
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The compaction fails when the Cleaner tried to remove a missing directory 
> from HDFS.
> {code:java}
> 2022-08-05 18:56:38,873 INFO org.apache.hadoop.hive.ql.txn.compactor.Cleaner: 
> [Cleaner-executor-thread-0]: Starting cleaning for 
> id:30,dbname:default,tableName:test_concur_compaction_minor,partName:null,state:�,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:4,errorMessage:null,workerId:
>  null,initiatorId: null 2022-08-05 18:56:38,888 ERROR 
> org.apache.hadoop.hive.ql.txn.compactor.Cleaner: [Cleaner-executor-thread-0]: 
> Caught exception when cleaning, unable to complete cleaning of 
> id:30,dbname:default,tableName:test_concur_compaction_minor,partName:null,state:�,type:MINOR,enqueueTime:0,start:0,properties:null,runAs:hive,tooManyAborts:false,hasOldAbort:false,highestWriteId:4,errorMessage:null,workerId:
>  null,initiatorId: null java.io.FileNotFoundException: File 
>

[jira] [Work logged] (HIVE-26480) Improve SYS.COMPACTIONS and SYS.TRANSACTIONS views

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26480?focusedWorklogId=803878=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803878
 ]

ASF GitHub Bot logged work on HIVE-26480:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 08:20
Start Date: 26/Aug/22 08:20
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on PR #3529:
URL: https://github.com/apache/hive/pull/3529#issuecomment-1228202468

   don't we have any q tests for the compaction sys tables/view?




Issue Time Tracking
---

Worklog Id: (was: 803878)
Time Spent: 1h  (was: 50m)

> Improve SYS.COMPACTIONS and SYS.TRANSACTIONS views
> --
>
> Key: HIVE-26480
> URL: https://issues.apache.org/jira/browse/HIVE-26480
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> * SYS.COMPACTIONS displays dates as millis since the epoch. We should replace 
> these with or add columns displaying dates in human-readable form.
>  * SYS.TRANSACTIONS tries to display human-readable dates, but they're wrong. 
> e.g. 53405-07-01 11:20:03 (probably is interpreting millis as seconds)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26480) Improve SYS.COMPACTIONS and SYS.TRANSACTIONS views

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26480?focusedWorklogId=803876=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803876
 ]

ASF GitHub Bot logged work on HIVE-26480:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 08:14
Start Date: 26/Aug/22 08:14
Worklog Time Spent: 10m 
  Work Description: deniskuzZ commented on PR #3529:
URL: https://github.com/apache/hive/pull/3529#issuecomment-1228195181

   @veghlaci05 can't find the schema script changes, just the upgrade




Issue Time Tracking
---

Worklog Id: (was: 803876)
Time Spent: 50m  (was: 40m)

> Improve SYS.COMPACTIONS and SYS.TRANSACTIONS views
> --
>
> Key: HIVE-26480
> URL: https://issues.apache.org/jira/browse/HIVE-26480
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: László Végh
>Assignee: László Végh
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> * SYS.COMPACTIONS displays dates as millis since the epoch. We should replace 
> these with or add columns displaying dates in human-readable form.
>  * SYS.TRANSACTIONS tries to display human-readable dates, but they're wrong. 
> e.g. 53405-07-01 11:20:03 (probably is interpreting millis as seconds)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Assigned] (HIVE-26487) Auto-compaction may hang or skip some of the eligible items

2022-08-26 Thread Denys Kuzmenko (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko reassigned HIVE-26487:
-

Assignee: Denys Kuzmenko

> Auto-compaction may hang or skip some of the eligible items 
> 
>
> Key: HIVE-26487
> URL: https://issues.apache.org/jira/browse/HIVE-26487
> Project: Hive
>  Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HIVE-26487) Auto-compaction may hang or skip some of the eligible items

2022-08-26 Thread Denys Kuzmenko (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Denys Kuzmenko resolved HIVE-26487.
---
Resolution: Fixed

Merged to master.
[~klcopp], thank you for the review!

> Auto-compaction may hang or skip some of the eligible items 
> 
>
> Key: HIVE-26487
> URL: https://issues.apache.org/jira/browse/HIVE-26487
> Project: Hive
>  Issue Type: Task
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26487) Auto-compaction may hang or skip some of the eligible items

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26487?focusedWorklogId=803873=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803873
 ]

ASF GitHub Bot logged work on HIVE-26487:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 07:56
Start Date: 26/Aug/22 07:56
Worklog Time Spent: 10m 
  Work Description: deniskuzZ merged PR #3537:
URL: https://github.com/apache/hive/pull/3537




Issue Time Tracking
---

Worklog Id: (was: 803873)
Time Spent: 20m  (was: 10m)

> Auto-compaction may hang or skip some of the eligible items 
> 
>
> Key: HIVE-26487
> URL: https://issues.apache.org/jira/browse/HIVE-26487
> Project: Hive
>  Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 20m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26400) Provide docker images for Hive

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=803864=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803864
 ]

ASF GitHub Bot logged work on HIVE-26400:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 07:30
Start Date: 26/Aug/22 07:30
Worklog Time Spent: 10m 
  Work Description: dengzhhu653 closed pull request #3448: HIVE-26400: 
Provide docker images for Hive
URL: https://github.com/apache/hive/pull/3448




Issue Time Tracking
---

Worklog Id: (was: 803864)
Time Spent: 2h 50m  (was: 2h 40m)

> Provide docker images for Hive
> --
>
> Key: HIVE-26400
> URL: https://issues.apache.org/jira/browse/HIVE-26400
> Project: Hive
>  Issue Type: Improvement
>  Components: Build Infrastructure
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Make Apache Hive be able to run inside docker container in pseudo-distributed 
> mode, with MySQL/Derby as its back database, provide the following:
>  * Quick-start/Debugging/Prepare a test env for Hive;
>  * Tools to build target image with specified version of Hive and its 
> dependencies;
>  * Images can be used as the basis for the Kubernetes operator.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-24969) Predicates may be removed when decorrelating subqueries with lateral

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-24969?focusedWorklogId=803856=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803856
 ]

ASF GitHub Bot logged work on HIVE-24969:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 07:01
Start Date: 26/Aug/22 07:01
Worklog Time Spent: 10m 
  Work Description: dengzhhu653 opened a new pull request, #3018:
URL: https://github.com/apache/hive/pull/3018

   
   
   ### What changes were proposed in this pull request?
   
   
   
   ### Why are the changes needed?
   
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   
   
   ### How was this patch tested?
   
   




Issue Time Tracking
---

Worklog Id: (was: 803856)
Time Spent: 3h 10m  (was: 3h)

> Predicates may be removed when decorrelating subqueries with lateral
> 
>
> Key: HIVE-24969
> URL: https://issues.apache.org/jira/browse/HIVE-24969
> Project: Hive
>  Issue Type: Bug
>  Components: Logical Optimizer
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> Step to reproduce：
> {code:java}
> select count(distinct logItem.triggerId)
> from service_stat_log LATERAL VIEW explode(logItems) LogItemTable AS logItem
> where logItem.dsp in ('delivery', 'ocpa')
> and logItem.iswin = true
> and logItem.adid in (
>  select distinct adId
>  from ad_info
>  where subAccountId in (16010, 14863));  {code}
> For predicates _logItem.dsp in ('delivery', 'ocpa')_  and _logItem.iswin = 
> true_ are removed when doing ppd: JOIN ->   RS  -> LVJ.  The JOIN has 
> candicates: logitem -> [logItem.dsp in ('delivery', 'ocpa'), logItem.iswin = 
> true]，when pushing them to the RS followed by LVJ,  none of them are pushed, 
> the candicates of logitem are removed finally by default, which cause to the 
> wrong result.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26046) MySQL's bit datatype is default to void datatype in hive

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26046?focusedWorklogId=803855=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803855
 ]

ASF GitHub Bot logged work on HIVE-26046:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 06:56
Start Date: 26/Aug/22 06:56
Worklog Time Spent: 10m 
  Work Description: dengzhhu653 commented on code in PR #3276:
URL: https://github.com/apache/hive/pull/3276#discussion_r955718045


##
ql/src/test/queries/clientpositive/dataconnector_mysql_bit_datatype.q:
##
@@ -0,0 +1,24 @@
+--!qt:database:mysql:q_test_mysql_datatype_mapping.sql
+

Issue Time Tracking
---

Worklog Id: (was: 803855)
Time Spent: 3h  (was: 2h 50m)

> MySQL's bit datatype is default to void datatype in hive
> 
>
> Key: HIVE-26046
> URL: https://issues.apache.org/jira/browse/HIVE-26046
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> describe on a table that contains a "bit" datatype gets mapped to void. We 
> need a explicit conversion logic in the MySQL ConnectorProvider to map it to 
> a suitable datatype in hive.
> {noformat}
> +---+---++
> |   col_name| data_type   
>   |  comment   |
> +---+---++
> | tbl_id| bigint  
>   | from deserializer  |
> | create_time   | int 
>   | from deserializer  |
> | db_id | bigint  
>   | from deserializer  |
> | last_access_time  | int 
>   | from deserializer  |
> | owner | varchar(767)
>   | from deserializer  |
> | owner_type| varchar(10) 
>   | from deserializer  |
> | retention | int 
>   | from deserializer  |
> | sd_id | bigint  
>   | from deserializer  |
> | tbl_name  | varchar(256)
>   | from deserializer  |
> | tbl_type  | varchar(128)
>   | from deserializer  |
> | view_expanded_text| string  
>   | from deserializer  |
> | view_original_text| string  
>   | from deserializer  |
> | is_rewrite_enabled| void
>   | from deserializer  |
> | write_id  | bigint  
>   | from deserializer  
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26046) MySQL's bit datatype is default to void datatype in hive

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26046?focusedWorklogId=803853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803853
 ]

ASF GitHub Bot logged work on HIVE-26046:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 06:53
Start Date: 26/Aug/22 06:53
Worklog Time Spent: 10m 
  Work Description: dengzhhu653 commented on code in PR #3276:
URL: https://github.com/apache/hive/pull/3276#discussion_r955715950


##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/dataconnector/jdbc/MySQLConnectorProvider.java:
##
@@ -90,10 +90,20 @@ protected String getDataType(String dbDataType, int size) {
 // map any db specific types here.
 switch (dbDataType.toLowerCase())
 {
+case "bit":
+  return toHiveBitType(size);
 default:
   mappedType = ColumnType.VOID_TYPE_NAME;
   break;
 }
 return mappedType;
   }
+
+  private String toHiveBitType(int size) {
+if (size <= 1) {
+  return ColumnType.BOOLEAN_TYPE_NAME;
+} else {
+  return ColumnType.BIGINT_TYPE_NAME;

Review Comment:
   Can we convert the bit(n>1) to hive binary type?





Issue Time Tracking
---

Worklog Id: (was: 803853)
Time Spent: 2h 50m  (was: 2h 40m)

> MySQL's bit datatype is default to void datatype in hive
> 
>
> Key: HIVE-26046
> URL: https://issues.apache.org/jira/browse/HIVE-26046
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Affects Versions: 4.0.0
>Reporter: Naveen Gangam
>Assignee: zhangbutao
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> describe on a table that contains a "bit" datatype gets mapped to void. We 
> need a explicit conversion logic in the MySQL ConnectorProvider to map it to 
> a suitable datatype in hive.
> {noformat}
> +---+---++
> |   col_name| data_type   
>   |  comment   |
> +---+---++
> | tbl_id| bigint  
>   | from deserializer  |
> | create_time   | int 
>   | from deserializer  |
> | db_id | bigint  
>   | from deserializer  |
> | last_access_time  | int 
>   | from deserializer  |
> | owner | varchar(767)
>   | from deserializer  |
> | owner_type| varchar(10) 
>   | from deserializer  |
> | retention | int 
>   | from deserializer  |
> | sd_id | bigint  
>   | from deserializer  |
> | tbl_name  | varchar(256)
>   | from deserializer  |
> | tbl_type  | varchar(128)
>   | from deserializer  |
> | view_expanded_text| string  
>   | from deserializer  |
> | view_original_text| string  
>   | from deserializer  |
> | is_rewrite_enabled| void
>   | from deserializer  |
> | write_id  | bigint  
>   | from deserializer  
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Work logged] (HIVE-26479) Add ability to set parameters for query-based compaction

2022-08-26 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/HIVE-26479?focusedWorklogId=803852=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-803852
 ]

ASF GitHub Bot logged work on HIVE-26479:
-

Author: ASF GitHub Bot
Created on: 26/Aug/22 06:41
Start Date: 26/Aug/22 06:41
Worklog Time Spent: 10m 
  Work Description: SourabhBadhya commented on code in PR #3528:
URL: https://github.com/apache/hive/pull/3528#discussion_r955707999


##
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/QueryCompactor.java:
##
@@ -272,5 +278,16 @@ static void removeFilesForMmTable(HiveConf conf, 
AcidDirectory dir) throws IOExc
 fs.delete(dead, true);
   }
 }
+
+static void overrideConfProps(HiveConf conf, CompactionInfo ci, 
Map properties) {
+  Stream.of(properties, ci.properties != null ? new 
StringableMap(ci.properties) : null)

Review Comment:
   Removed the following file - 
https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/StringableMap.java

   However, regarding the behaviour of toString wherein null is given a length 
of -1. I have retained the same behaviour moving forward mainly because of 
[HIVE-17391](https://issues.apache.org/jira/browse/HIVE-17391).





Issue Time Tracking
---

Worklog Id: (was: 803852)
Time Spent: 3h  (was: 2h 50m)

> Add ability to set parameters for query-based compaction
> 
>
> Key: HIVE-26479
> URL: https://issues.apache.org/jira/browse/HIVE-26479
> Project: Hive
>  Issue Type: Improvement
>Reporter: Sourabh Badhya
>Assignee: Sourabh Badhya
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354] introduced the 
> ability to set some parameters for the compaction through table properties, 
> like the mapper memory size or compaction thresholds. This could be useful 
> for the query-based compaction as well, for example if the insert of the 
> query-based compaction is failing, we would have a possibility to tune the 
> compaction run directly. First it should be investigated which properties are 
> possible and would make sense to set for the query base compaction. Then 
> implement this feature for the query-based compaction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)