[jira] [Work logged] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?focusedWorklogId=823264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823264 ] ASF GitHub Bot logged work on HIVE-26705: - Author: ASF GitHub Bot Created on: 04/Nov/22 05:25 Start Date: 04/Nov/22 05:25 Worklog Time Spent: 10m Work Description: slfan1989 commented on PR #3729: URL: https://github.com/apache/hive/pull/3729#issuecomment-1302994124 > LGTM @ayushtkn Thank you very much for your help reviewing the code and merging the pr! Issue Time Tracking --- Worklog Id: (was: 823264) Time Spent: 1h (was: 50m) > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?focusedWorklogId=823262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823262 ] ASF GitHub Bot logged work on HIVE-26705: - Author: ASF GitHub Bot Created on: 04/Nov/22 05:24 Start Date: 04/Nov/22 05:24 Worklog Time Spent: 10m Work Description: slfan1989 commented on PR #3729: URL: https://github.com/apache/hive/pull/3729#issuecomment-1302993587 > +1 (non-binding) > > Thank you, @slfan1989 ! @cnauroth Thank you very much for your help reviewing the code! Issue Time Tracking --- Worklog Id: (was: 823262) Time Spent: 50m (was: 40m) > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-25483) TxnHandler::acquireLock should close the DB conn to avoid connection leaks
[
https://issues.apache.org/jira/browse/HIVE-25483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KIRTI RUGE reassigned HIVE-25483:
-
Assignee: KIRTI RUGE
> TxnHandler::acquireLock should close the DB conn to avoid connection leaks
> --
>
> Key: HIVE-25483
> URL: https://issues.apache.org/jira/browse/HIVE-25483
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: KIRTI RUGE
>Priority: Major
>
> TxnHandler::acquireLock should close DB connection on exiting the function.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5688]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5726]
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java#L5737-L5740]
> If there are any exceptions downstream, this connection isn't closed
> cleanly. In a corner case, hikari connection leak detector reported the
> following
> {noformat}
> 2021-08-26 09:19:18,102 WARN com.zaxxer.hikari.pool.ProxyLeakTask:
> [HikariPool-4 housekeeper]: Connection leak detection triggered for
> org.postgresql.jdbc.PgConnection@77f76747, stack trace follows
> java.lang.Exception: Apparent connection leak detected
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.getDbConn(TxnHandler.java:3843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.acquireLock(TxnHandler.java:5135)
>
> at
> org.apache.hadoop.hive.ql.txn.compactor.Initiator.run(Initiator.java:107)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26085) "getTableObjectByName method should ignore it" exception doesn't include cause
[
https://issues.apache.org/jira/browse/HIVE-26085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
KIRTI RUGE reassigned HIVE-26085:
-
Assignee: KIRTI RUGE
> "getTableObjectByName method should ignore it" exception doesn't include cause
> --
>
> Key: HIVE-26085
> URL: https://issues.apache.org/jira/browse/HIVE-26085
> Project: Hive
> Issue Type: Improvement
> Components: Parser
> Environment: HDP 3.1.5
>Reporter: Wataru Yukawa
>Assignee: KIRTI RUGE
>Priority: Major
>
> current logic doesn't contain cause
> https://github.com/apache/hive/blob/a6e93633dc15aba179fb6ad422be4cbc88adf071/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L12208
> but
> {code}
> throw new SemanticException("Got exception though getTableObjectByName method
> should ignore it", e)
> {code}
> seems better for troubleshooting.
> We encounter this issue when user access hive view where user doesn't hive
> permission in original hive table.
> {code}
> create view aaa_view
> as
> select ... from aaa
> {code}
> "getTableObjectByName" exception happens when user try to access "aaa_view"
> but doesn't have permission "aaa" table with apache ranger.
> {code}
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.parse.SemanticException:Got exception though
> getTableObjectByName method should ignore it
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.walkASTMarkTABREF(SemanticAnalyzer.java:12020)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.rewriteASTWithMaskAndFilter(SemanticAnalyzer.java:12139)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.replaceViewReferenceWithDefinition(SemanticAnalyzer.java:2608)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2192)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2103)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2257)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2088)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12234)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12328)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:367)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:290)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1870)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1817)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1812)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
> ... 26 common frames omitted
> {code}
> In this case, we can't see error log like "permission denied" when user try
> to access "aaa_view".
> So, it would be nice to add cause
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17628671#comment-17628671 ] Ayush Saxena commented on HIVE-26705: - Committed to master. Thanx [~slfan1989] for the contribution and [~cnauroth] for the review!!! > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena updated HIVE-26705: Fix Version/s: 4.0.0 > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-26705. - Resolution: Fixed > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?focusedWorklogId=823258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823258 ] ASF GitHub Bot logged work on HIVE-26705: - Author: ASF GitHub Bot Created on: 04/Nov/22 04:47 Start Date: 04/Nov/22 04:47 Worklog Time Spent: 10m Work Description: ayushtkn merged PR #3729: URL: https://github.com/apache/hive/pull/3729 Issue Time Tracking --- Worklog Id: (was: 823258) Time Spent: 40m (was: 0.5h) > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?focusedWorklogId=823236=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823236 ] ASF GitHub Bot logged work on HIVE-26705: - Author: ASF GitHub Bot Created on: 04/Nov/22 02:42 Start Date: 04/Nov/22 02:42 Worklog Time Spent: 10m Work Description: slfan1989 commented on PR #3729: URL: https://github.com/apache/hive/pull/3729#issuecomment-1302884961 @ayushtkn Can you help review this pr? Thank you very much! Issue Time Tracking --- Worklog Id: (was: 823236) Time Spent: 0.5h (was: 20m) > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?focusedWorklogId=823222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823222 ] ASF GitHub Bot logged work on HIVE-26705: - Author: ASF GitHub Bot Created on: 04/Nov/22 00:32 Start Date: 04/Nov/22 00:32 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3729: URL: https://github.com/apache/hive/pull/3729#issuecomment-1302822301 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3729) [](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3729=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3729=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3729=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3729=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3729=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3729=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823222) Time Spent: 20m (was: 10m) > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26513) Hive Metastore fail to start in Hadoop3.3.x when Kerberos enabled
[
https://issues.apache.org/jira/browse/HIVE-26513?focusedWorklogId=823220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823220
]
ASF GitHub Bot logged work on HIVE-26513:
-
Author: ASF GitHub Bot
Created on: 04/Nov/22 00:25
Start Date: 04/Nov/22 00:25
Worklog Time Spent: 10m
Work Description: github-actions[bot] commented on PR #3573:
URL: https://github.com/apache/hive/pull/3573#issuecomment-1302819013
This pull request has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the d...@hive.apache.org list if the patch is in
need of reviews.
Issue Time Tracking
---
Worklog Id: (was: 823220)
Time Spent: 20m (was: 10m)
> Hive Metastore fail to start in Hadoop3.3.x when Kerberos enabled
> -
>
> Key: HIVE-26513
> URL: https://issues.apache.org/jira/browse/HIVE-26513
> Project: Hive
> Issue Type: Bug
> Environment: Hive-3.1.3
> Hadoop-3.3.4
> Zookeeper-3.5.9
>Reporter: Yu Hou
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Fail to start Hive metastore when enable Kerberos in Hadoop 3.3.x
> {code:java}
> 2022-09-02T06:20:04,458 INFO [main]: zookeeper.ZooKeeper
> (ZooKeeper.java:(868)) - Initiating client connection,
> connectString=ambari-agent-01:2181,ambari-agent-02:2181,ambari-server:2181
> sessionTimeout=6 watcher=org.apache.curator.ConnectionState@433348bc
> 2022-09-02T06:20:04,470 INFO [main]: common.X509Util
> (X509Util.java:(79)) - Setting -D
> jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated
> TLS renegotiation
> 2022-09-02T06:20:04,499 INFO [main]: zookeeper.ClientCnxnSocket
> (ClientCnxnSocket.java:initProperties(237)) - jute.maxbuffer value is 4194304
> Bytes
> 2022-09-02T06:20:04,540 INFO [main]: zookeeper.ClientCnxn
> (ClientCnxn.java:initRequestTimeout(1653)) - zookeeper.request.timeout value
> is 0. feature enabled=
> 2022-09-02T06:20:04,587 ERROR [main]: metastore.HiveMetaStore
> (HiveMetaStore.java:startMetaStore(9056)) - java.lang.NoSuchMethodError:
> org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModeStatPathAndBytesable;
> at
> org.apache.hadoop.hive.metastore.security.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:158)
> at
> org.apache.hadoop.hive.metastore.security.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:234)
> at
> org.apache.hadoop.hive.metastore.security.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:471)
> at
> org.apache.hadoop.hive.metastore.security.MetastoreDelegationTokenManager.startDelegationTokenSecretManager(MetastoreDelegationTokenManager.java:76)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8952)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8854)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
> 2022-09-02T06:20:04,613 INFO [main-SendThread(ambari-agent-01:2181)]:
> zookeeper.Login (Login.java:login(302)) - HiveZooKeeperClient successfully
> logged in.
> 2022-09-02T06:20:04,615 INFO [Thread-6]: zookeeper.Login
> (Login.java:run(135)) - TGT refresh thread started.
> 2022-09-02T06:20:04,640 INFO [main-SendThread(ambari-agent-01:2181)]:
> client.ZooKeeperSaslClient (SecurityUtils.java:run(128)) - Client will use
> GSSAPI as SASL mechanism.
> 2022-09-02T06:20:04,588 ERROR [main]: metastore.HiveMetaStore
> (HiveMetaStore.java:main(8859)) - Metastore Thrift Server threw an
> exception...
> java.lang.NoSuchMethodError:
> org.apache.curator.framework.api.CreateBuilder.creatingParentsIfNeeded()Lorg/apache/curator/framework/api/ProtectACLCreateModeStatPathAndBytesable;
> at
> org.apache.hadoop.hive.metastore.security.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:158)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.metastore.security.ZooKeeperTokenStore.initClientAndPaths(ZooKeeperTokenStore.java:234)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
> org.apache.hadoop.hive.metastore.security.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:471)
> ~[hive-exec-3.1.3.jar:3.1.3]
> at
>
[jira] [Work logged] (HIVE-26012) HMS APIs to be enhanced for metadata replication
[ https://issues.apache.org/jira/browse/HIVE-26012?focusedWorklogId=823221=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823221 ] ASF GitHub Bot logged work on HIVE-26012: - Author: ASF GitHub Bot Created on: 04/Nov/22 00:25 Start Date: 04/Nov/22 00:25 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #3477: HIVE-26012: HMS APIs to be enhanced for metadata replication URL: https://github.com/apache/hive/pull/3477 Issue Time Tracking --- Worklog Id: (was: 823221) Time Spent: 6h (was: 5h 50m) > HMS APIs to be enhanced for metadata replication > > > Key: HIVE-26012 > URL: https://issues.apache.org/jira/browse/HIVE-26012 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 3.1.0 >Reporter: Naveen Gangam >Assignee: Hongdan Zhu >Priority: Major > Labels: pull-request-available > Attachments: HMS APIs to be enhanced for metadata replication.docx > > Time Spent: 6h > Remaining Estimate: 0h > > HMS currently has APIs like these that automatically create/delete the > directories on the associated DFS. > [create/drop]_database > [create/drop]_table* > [add/append/drop]_partition* > This is expected and should be this way when query processors use this APIs. > However, when tools that replicate hive metadata use this APIs on the target > cluster, creating these dirs on target side which cause the replication of > DFS-snapshots to fail. > So we if provide an option to bypass this creation of dirs, dfs replications > will be smoother. In the future we will need to restrict users that can use > these APIs. So we will have some sort of an authorization policy. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?focusedWorklogId=823218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823218 ] ASF GitHub Bot logged work on HIVE-26705: - Author: ASF GitHub Bot Created on: 03/Nov/22 23:40 Start Date: 03/Nov/22 23:40 Worklog Time Spent: 10m Work Description: slfan1989 opened a new pull request, #3729: URL: https://github.com/apache/hive/pull/3729 JIRA: HIVE-26705. Fix Typo in hive-kudu-handler. When reading the code, I found typo and repaired the typo of the moudle. Issue Time Tracking --- Worklog Id: (was: 823218) Remaining Estimate: 0h Time Spent: 10m > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26705: -- Labels: pull-request-available (was: ) > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun reassigned HIVE-26705: > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-26705) Fix Typo in hive-kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-26705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26705 started by fanshilun. > Fix Typo in hive-kudu-handler > - > > Key: HIVE-26705 > URL: https://issues.apache.org/jira/browse/HIVE-26705 > Project: Hive > Issue Type: Improvement >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > > When reading the code, I found typo and repaired the typo of the moudle. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823200=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823200 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 21:13 Start Date: 03/Nov/22 21:13 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3727: URL: https://github.com/apache/hive/pull/3727#issuecomment-1302668666 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3727) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3727=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3727=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823200) Time Spent: 1h 20m (was: 1h 10m) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26704) Cleaner shouldn't be blocked by global min open txnId
[ https://issues.apache.org/jira/browse/HIVE-26704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26704: -- Labels: pull-request-available (was: ) > Cleaner shouldn't be blocked by global min open txnId > - > > Key: HIVE-26704 > URL: https://issues.apache.org/jira/browse/HIVE-26704 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26704) Cleaner shouldn't be blocked by global min open txnId
[ https://issues.apache.org/jira/browse/HIVE-26704?focusedWorklogId=823192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823192 ] ASF GitHub Bot logged work on HIVE-26704: - Author: ASF GitHub Bot Created on: 03/Nov/22 20:39 Start Date: 03/Nov/22 20:39 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3576: URL: https://github.com/apache/hive/pull/3576#issuecomment-1302638546 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3576) [](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3576=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3576=false=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3576=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=CODE_SMELL) [18 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3576=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3576=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3576=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823192) Remaining Estimate: 0h Time Spent: 10m > Cleaner shouldn't be blocked by global min open txnId > - > > Key: HIVE-26704 > URL: https://issues.apache.org/jira/browse/HIVE-26704 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823160=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823160 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 18:05 Start Date: 03/Nov/22 18:05 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3727: URL: https://github.com/apache/hive/pull/3727#issuecomment-1302487087 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3727) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3727=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3727=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823160) Time Spent: 1h 10m (was: 1h) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26702) Backport HIVE-17315 (more flexible metastore database connection pooling) to 3.2.0.
[
https://issues.apache.org/jira/browse/HIVE-26702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HIVE-26702:
-
Description: HIVE-17315 added support for more flexible metastore
connection pooling configuration, integration with DBCP and an important
capability to the HikariCP integration: passthrough configuration of any
additional HikariCP configurations, such as {{{}minimumIdle{}}}. This issue
proposes to backport all 5 sub-tasks of HIVE-17315 for inclusion in the
upcoming 3.2.0 release. (was: HIVE-17317 added support for using DBCP as the
metastore's database connection pooling implementation. It also added an
important capability to the HikariCP integration: passthrough configuration of
any additional HikariCP configurations, such as {{{}minimumIdle{}}}. This issue
proposes to backport this for the upcoming 3.2.0 release.)
Summary: Backport HIVE-17315 (more flexible metastore database
connection pooling) to 3.2.0. (was: Backport HIVE-17317 (metastore DBCP and
HikariCP configuration support) to 3.x.)
> Backport HIVE-17315 (more flexible metastore database connection pooling) to
> 3.2.0.
> ---
>
> Key: HIVE-26702
> URL: https://issues.apache.org/jira/browse/HIVE-26702
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
>
> HIVE-17315 added support for more flexible metastore connection pooling
> configuration, integration with DBCP and an important capability to the
> HikariCP integration: passthrough configuration of any additional HikariCP
> configurations, such as {{{}minimumIdle{}}}. This issue proposes to backport
> all 5 sub-tasks of HIVE-17315 for inclusion in the upcoming 3.2.0 release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26702) Backport HIVE-17317 (metastore DBCP and HikariCP configuration support) to 3.x.
[
https://issues.apache.org/jira/browse/HIVE-26702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reassigned HIVE-26702:
> Backport HIVE-17317 (metastore DBCP and HikariCP configuration support) to
> 3.x.
> ---
>
> Key: HIVE-26702
> URL: https://issues.apache.org/jira/browse/HIVE-26702
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
>
> HIVE-17317 added support for using DBCP as the metastore's database
> connection pooling implementation. It also added an important capability to
> the HikariCP integration: passthrough configuration of any additional
> HikariCP configurations, such as {{{}minimumIdle{}}}. This issue proposes to
> backport this for the upcoming 3.2.0 release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823149=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823149 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 17:21 Start Date: 03/Nov/22 17:21 Worklog Time Spent: 10m Work Description: cnauroth commented on PR #3727: URL: https://github.com/apache/hive/pull/3727#issuecomment-1302435776 Something odd happened in that last pre-commit, and it looks unrelated to this change. I triggered a re-run: http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-3727/2/pipeline/ Issue Time Tracking --- Worklog Id: (was: 823149) Time Spent: 1h (was: 50m) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26684) Upgrade maven-shade-plugin from 3.4.1 for bug fixes.
[ https://issues.apache.org/jira/browse/HIVE-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17628426#comment-17628426 ] Chris Nauroth commented on HIVE-26684: -- Thank you, [~ngangam] and [~ayushtkn] ! > Upgrade maven-shade-plugin from 3.4.1 for bug fixes. > > > Key: HIVE-26684 > URL: https://issues.apache.org/jira/browse/HIVE-26684 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The Hive build currently runs with maven-shade-plugin version 3.1.1, released > in April 2018. This issue proposes to upgrade to the latest version, 3.4.1, > released in October 2022. See HIVE-26648 for an example of another patch that > is blocked due to a bug in version 3.1.1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-13288) Confusing exception message in DagUtils.localizeResource
[
https://issues.apache.org/jira/browse/HIVE-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17628425#comment-17628425
]
Chris Nauroth commented on HIVE-13288:
--
Thank you, [~rbalamohan] and [~abstractdog] !
> Confusing exception message in DagUtils.localizeResource
>
>
> Key: HIVE-13288
> URL: https://issues.apache.org/jira/browse/HIVE-13288
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Jeff Zhang
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I got the following exception when query through hive server2. And check the
> source code, it it due to some error when copying data from local to hdfs.
> But the IOException is ignored and assume that it is due to another thread is
> also writing. I don't think it make sense to assume that, at least should log
> the IOException.
> {code}
> LOG.info("Localizing resource because it does not exist: " + src + " to dest:
> " + dest);
> try {
> destFS.copyFromLocalFile(false, false, src, dest);
> } catch (IOException e) {
> LOG.info("Looks like another thread is writing the same file will
> wait.");
> int waitAttempts =
>
> conf.getInt(HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.varname,
>
> HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.defaultIntVal);
> long sleepInterval = HiveConf.getTimeVar(
> conf, HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL,
> TimeUnit.MILLISECONDS);
> LOG.info("Number of wait attempts: " + waitAttempts + ". Wait
> interval: "
> + sleepInterval);
> boolean found = false;
> {code}
> {noformat}
> 2016-03-15 11:25:39,921 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:getHiveJarDirectory(876)) - Jar dir is
> null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jeff/.hiveJars
> 2016-03-15 11:25:40,058 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(952)) - Localizing resource
> because it does not exist:
> file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to dest:
> hdfs://sandbox.hortonworks.com:8020/user/jeff/.hiveJars/hive-exec-1.2.1.2.3.2.0-2950-a97c953db414a4f792d868e2b0417578a61ccfa368048016926117b641b07f34.jar
> 2016-03-15 11:25:40,063 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(956)) - Looks like another
> thread is writing the same file will wait.
> 2016-03-15 11:25:40,064 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(963)) - Number of wait attempts:
> 5. Wait interval: 5000
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Shutting down
> the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Metastore
> shutdown complete.
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 2016-03-15 11:25:53,573 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/e43fbaab-a659-4331-90cb-0ea0b2098e25_resources
> 2016-03-15 11:25:53,577 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory: /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,582 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/hive/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,587 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory:
>
[jira] [Work logged] (HIVE-26693) HS2 can not read/write hive_catalog iceberg table created by other engines
[
https://issues.apache.org/jira/browse/HIVE-26693?focusedWorklogId=823121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823121
]
ASF GitHub Bot logged work on HIVE-26693:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 15:50
Start Date: 03/Nov/22 15:50
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3726:
URL: https://github.com/apache/hive/pull/3726#issuecomment-1302314882
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3726)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3726=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3726=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3726=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3726=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3726=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823121)
Time Spent: 40m (was: 0.5h)
> HS2 can not read/write hive_catalog iceberg table created by other engines
> --
>
> Key: HIVE-26693
> URL: https://issues.apache.org/jira/browse/HIVE-26693
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, StorageHandler
>Affects Versions: 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Step to reproduce:
> # Create hive_catalog iceberg table by Trino/Presto/Flink (Spark with
> _iceberg.engine.hive.enabled_ disabled)
> # show table info with hive beeline:
> {code:java}
> ++
> | createtab_stmt |
> ++
> | CREATE EXTERNAL TABLE `iceberg_hive`.`testtrinoice`( |
> | `id` int) |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.mapred.FileInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.mapred.FileOutputFormat' |
> | LOCATION
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=823101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823101
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 14:18
Start Date: 03/Nov/22 14:18
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3724:
URL: https://github.com/apache/hive/pull/3724#issuecomment-1302183767
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3724)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3724=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3724=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823101)
Time Spent: 2h (was: 1h 50m)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
>
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823099 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 14:06 Start Date: 03/Nov/22 14:06 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3727: URL: https://github.com/apache/hive/pull/3727#issuecomment-1302167306 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3727) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3727=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3727=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3727=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3727=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823099) Time Spent: 50m (was: 40m) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26243) Add vectorized implementation of the 'ds_kll_sketch' UDAF
[
https://issues.apache.org/jira/browse/HIVE-26243?focusedWorklogId=823078=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823078
]
ASF GitHub Bot logged work on HIVE-26243:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 13:00
Start Date: 03/Nov/22 13:00
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3317:
URL: https://github.com/apache/hive/pull/3317#discussion_r1012869267
##
standalone-metastore/pom.xml:
##
@@ -181,6 +182,17 @@
commons-lang3
${commons-lang3.version}
+
+org.apache.datasketches
Review Comment:
why do you need this, it's already defined in root pom DM?
Issue Time Tracking
---
Worklog Id: (was: 823078)
Time Spent: 3h 10m (was: 3h)
> Add vectorized implementation of the 'ds_kll_sketch' UDAF
> -
>
> Key: HIVE-26243
> URL: https://issues.apache.org/jira/browse/HIVE-26243
> Project: Hive
> Issue Type: Improvement
> Components: UDF, Vectorization
>Affects Versions: 4.0.0-alpha-2
>Reporter: Alessandro Solimando
>Assignee: Alessandro Solimando
>Priority: Major
> Labels: pull-request-available
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> _ds_kll_sketch_ UDAF does not have a vectorized implementation at the moment,
> the present ticket aims at bridging this gap.
> This is particularly important because vectorization has an "all or nothing"
> approach, so if this function is used at the side of vectorized functions,
> they won't be able to benefit from vectorized execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-26243) Add vectorized implementation of the 'ds_kll_sketch' UDAF
[
https://issues.apache.org/jira/browse/HIVE-26243?focusedWorklogId=823077=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823077
]
ASF GitHub Bot logged work on HIVE-26243:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 12:59
Start Date: 03/Nov/22 12:59
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3317:
URL: https://github.com/apache/hive/pull/3317#discussion_r1012845371
##
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java:
##
@@ -304,7 +327,7 @@ public enum EnabledOverride {
nameMap.put(
vectorizationEnabledOverride.name().toLowerCase(),
vectorizationEnabledOverride);
Review Comment:
could you please create toString() method that returns name().toLowerCase()
and use that
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/common/histogram/KllHistogramEstimator.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.common.histogram;
+
+import org.apache.datasketches.kll.KllFloatsSketch;
+import org.apache.hadoop.hive.common.histogram.kll.KllUtils;
+import org.apache.hadoop.hive.common.type.HiveDecimal;
+import org.apache.hadoop.hive.ql.util.JavaDataModel;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+
+public class KllHistogramEstimator {
Review Comment:
should we keep those classes in hive-exec module?
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/common/histogram/KllHistogramEstimatorFactory.java:
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.common.histogram;
+
+import org.apache.hadoop.hive.common.histogram.kll.KllUtils;
+
+public class KllHistogramEstimatorFactory {
+
+ private KllHistogramEstimatorFactory() {
+throw new AssertionError("Suppress default constructor for non
instantiation");
+ }
+
+ /**
+ * This function deserializes the serialized KLL histogram estimator from a
byte array.
+ * @param buf to deserialize
+ * @return KLL histogram estimator
+ */
+ public static KllHistogramEstimator getKllHistogramEstimator(byte[] buf) {
Review Comment:
is it used?
##
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/common/histogram/KllHistogramEstimatorFactory.java:
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.hive.common.histogram;
+
+import
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823076=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823076 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 12:57 Start Date: 03/Nov/22 12:57 Worklog Time Spent: 10m Work Description: devaspatikrishnatri commented on PR #3727: URL: https://github.com/apache/hive/pull/3727#issuecomment-1302057676 @ayushtkn @nrg4878 @cnauroth please see this Issue Time Tracking --- Worklog Id: (was: 823076) Time Spent: 40m (was: 0.5h) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823075 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 12:56 Start Date: 03/Nov/22 12:56 Worklog Time Spent: 10m Work Description: devaspatikrishnatri opened a new pull request, #3727: URL: https://github.com/apache/hive/pull/3727 > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26648) Upgrade Bouncy Castle to 1.70 due to high CVEs
[ https://issues.apache.org/jira/browse/HIVE-26648?focusedWorklogId=823073=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823073 ] ASF GitHub Bot logged work on HIVE-26648: - Author: ASF GitHub Bot Created on: 03/Nov/22 12:48 Start Date: 03/Nov/22 12:48 Worklog Time Spent: 10m Work Description: devaspatikrishnatri closed pull request #3688: HIVE-26648:Upgrade Bouncy Castle to 1.70 due to high CVEs URL: https://github.com/apache/hive/pull/3688 Issue Time Tracking --- Worklog Id: (was: 823073) Time Spent: 20m (was: 10m) > Upgrade Bouncy Castle to 1.70 due to high CVEs > --- > > Key: HIVE-26648 > URL: https://issues.apache.org/jira/browse/HIVE-26648 > Project: Hive > Issue Type: Task >Reporter: Devaspati Krishnatri >Assignee: Devaspati Krishnatri >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26693) HS2 can not read/write hive_catalog iceberg table created by other engines
[
https://issues.apache.org/jira/browse/HIVE-26693?focusedWorklogId=823051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823051
]
ASF GitHub Bot logged work on HIVE-26693:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 11:52
Start Date: 03/Nov/22 11:52
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3726:
URL: https://github.com/apache/hive/pull/3726#issuecomment-1301988672
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3726)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3726=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3726=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3726=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3726=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3726=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3726=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 823051)
Time Spent: 0.5h (was: 20m)
> HS2 can not read/write hive_catalog iceberg table created by other engines
> --
>
> Key: HIVE-26693
> URL: https://issues.apache.org/jira/browse/HIVE-26693
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, StorageHandler
>Affects Versions: 4.0.0-alpha-2
>Reporter: zhangbutao
>Assignee: zhangbutao
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Step to reproduce:
> # Create hive_catalog iceberg table by Trino/Presto/Flink (Spark with
> _iceberg.engine.hive.enabled_ disabled)
> # show table info with hive beeline:
> {code:java}
> ++
> | createtab_stmt |
> ++
> | CREATE EXTERNAL TABLE `iceberg_hive`.`testtrinoice`( |
> | `id` int) |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.mapred.FileInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.mapred.FileOutputFormat' |
> | LOCATION
[jira] [Work logged] (HIVE-25790) Make managed table copies handle updates (FileUtils)
[
https://issues.apache.org/jira/browse/HIVE-25790?focusedWorklogId=823024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823024
]
ASF GitHub Bot logged work on HIVE-25790:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 10:36
Start Date: 03/Nov/22 10:36
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3582:
URL: https://github.com/apache/hive/pull/3582#discussion_r1012655098
##
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java:
##
@@ -492,20 +495,94 @@ private void doRegularCopyOnce(FileSystem sourceFs,
List srcList,
throw new IOException(e);
}
} else {
- //Destination should be empty
+ // Destination should have identical files only
if (overWrite) {
-deleteSubDirs(destinationFs, destination);
+leaveIdenticalFilesOnly(sourceFs, paths, destinationFs, destination);
}
copyFilesBetweenFS(sourceFs, paths, destinationFs, destination, false,
true);
}
}
- private void deleteSubDirs(FileSystem fs, Path path) throws IOException {
-//Delete the root path instead of doing a listing
-//This is more optimised
-delete(fs, path, true);
-//Recreate just the Root folder
-mkdirs(fs, path);
+ /**
+ * Leave identical files only between source and destination. It gets source
paths.
+ */
+ void leaveIdenticalFilesOnly(FileSystem sourceFs, Path[] srcPaths,
+ FileSystem destinationFs, Path destinationPath) throws IOException {
+for (Path srcPath : srcPaths) {
+ leaveIdenticalFilesOnlyFileByFile(sourceFs, srcPath,
+ destinationFs, new Path(destinationPath, srcPath.getName()));
+}
+ }
+
+ /**
+ * Leave identical files only between source and destination. It gets a
source path.
+ */
+ private void leaveIdenticalFilesOnlyFileByFile(FileSystem sourceFs, Path
srcPath,
+ FileSystem destinationFs, Path dstPath) throws IOException {
+boolean toDelete = false;
+Type dstType = getType(destinationFs, dstPath);
+Type srcType = getType(sourceFs, srcPath);
+LOG.debug("Source path: {}, type: {}", srcPath, srcType);
+LOG.debug("Destination path: {}, type: {}", dstPath, dstType);
+switch (dstType) {
+case NONE:
+ // If destination is none, then no need to delete anything.
+ break;
+case FILE:
+ // If destination is a file,
+ if (srcType == Type.FILE) {
+// If source is a file, then delete the destination file if it is not
same as source.
+toDelete = !FileUtils.isSameFile(sourceFs, srcPath, destinationFs,
dstPath);
+LOG.debug("Same file?: {}", !toDelete);
+ } else {
+// Otherwise, delete the destination file.
+toDelete = true;
+ }
+ break;
+case DIRECTORY:
+ // If destination is a directory,
+ if (srcType == Type.DIRECTORY) {
+// If both are directories, visit for children of both.
+Set bothChildNames = Stream.concat(
+Arrays.stream(sourceFs.listStatus(srcPath))
Review Comment:
you are listing plenty of times: here and when calling getType, could this
be optimized?
Issue Time Tracking
---
Worklog Id: (was: 823024)
Time Spent: 1h 20m (was: 1h 10m)
> Make managed table copies handle updates (FileUtils)
>
>
> Key: HIVE-25790
> URL: https://issues.apache.org/jira/browse/HIVE-25790
> Project: Hive
> Issue Type: Sub-task
>Reporter: Haymant Mangla
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (HIVE-25790) Make managed table copies handle updates (FileUtils)
[
https://issues.apache.org/jira/browse/HIVE-25790?focusedWorklogId=823022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823022
]
ASF GitHub Bot logged work on HIVE-25790:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 10:35
Start Date: 03/Nov/22 10:35
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3582:
URL: https://github.com/apache/hive/pull/3582#discussion_r1012643637
##
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java:
##
@@ -492,20 +495,94 @@ private void doRegularCopyOnce(FileSystem sourceFs,
List srcList,
throw new IOException(e);
}
} else {
- //Destination should be empty
+ // Destination should have identical files only
if (overWrite) {
-deleteSubDirs(destinationFs, destination);
+leaveIdenticalFilesOnly(sourceFs, paths, destinationFs, destination);
}
copyFilesBetweenFS(sourceFs, paths, destinationFs, destination, false,
true);
}
}
- private void deleteSubDirs(FileSystem fs, Path path) throws IOException {
-//Delete the root path instead of doing a listing
-//This is more optimised
-delete(fs, path, true);
-//Recreate just the Root folder
-mkdirs(fs, path);
+ /**
+ * Leave identical files only between source and destination. It gets source
paths.
+ */
+ void leaveIdenticalFilesOnly(FileSystem sourceFs, Path[] srcPaths,
+ FileSystem destinationFs, Path destinationPath) throws IOException {
+for (Path srcPath : srcPaths) {
+ leaveIdenticalFilesOnlyFileByFile(sourceFs, srcPath,
+ destinationFs, new Path(destinationPath, srcPath.getName()));
+}
+ }
+
+ /**
+ * Leave identical files only between source and destination. It gets a
source path.
+ */
+ private void leaveIdenticalFilesOnlyFileByFile(FileSystem sourceFs, Path
srcPath,
+ FileSystem destinationFs, Path dstPath) throws IOException {
+boolean toDelete = false;
+Type dstType = getType(destinationFs, dstPath);
+Type srcType = getType(sourceFs, srcPath);
+LOG.debug("Source path: {}, type: {}", srcPath, srcType);
+LOG.debug("Destination path: {}, type: {}", dstPath, dstType);
+switch (dstType) {
+case NONE:
+ // If destination is none, then no need to delete anything.
+ break;
+case FILE:
+ // If destination is a file,
+ if (srcType == Type.FILE) {
+// If source is a file, then delete the destination file if it is not
same as source.
+toDelete = !FileUtils.isSameFile(sourceFs, srcPath, destinationFs,
dstPath);
+LOG.debug("Same file?: {}", !toDelete);
+ } else {
+// Otherwise, delete the destination file.
+toDelete = true;
Review Comment:
is it possible that srcType and dstType would be different?
##
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java:
##
@@ -492,20 +495,94 @@ private void doRegularCopyOnce(FileSystem sourceFs,
List srcList,
throw new IOException(e);
}
} else {
- //Destination should be empty
+ // Destination should have identical files only
if (overWrite) {
-deleteSubDirs(destinationFs, destination);
+leaveIdenticalFilesOnly(sourceFs, paths, destinationFs, destination);
}
copyFilesBetweenFS(sourceFs, paths, destinationFs, destination, false,
true);
}
}
- private void deleteSubDirs(FileSystem fs, Path path) throws IOException {
-//Delete the root path instead of doing a listing
-//This is more optimised
-delete(fs, path, true);
-//Recreate just the Root folder
-mkdirs(fs, path);
+ /**
+ * Leave identical files only between source and destination. It gets source
paths.
+ */
+ void leaveIdenticalFilesOnly(FileSystem sourceFs, Path[] srcPaths,
+ FileSystem destinationFs, Path destinationPath) throws IOException {
+for (Path srcPath : srcPaths) {
+ leaveIdenticalFilesOnlyFileByFile(sourceFs, srcPath,
+ destinationFs, new Path(destinationPath, srcPath.getName()));
+}
+ }
+
+ /**
+ * Leave identical files only between source and destination. It gets a
source path.
+ */
+ private void leaveIdenticalFilesOnlyFileByFile(FileSystem sourceFs, Path
srcPath,
+ FileSystem destinationFs, Path dstPath) throws IOException {
+boolean toDelete = false;
+Type dstType = getType(destinationFs, dstPath);
+Type srcType = getType(sourceFs, srcPath);
+LOG.debug("Source path: {}, type: {}", srcPath, srcType);
+LOG.debug("Destination path: {}, type: {}", dstPath, dstType);
+switch (dstType) {
+case NONE:
+ // If destination is none, then no need to delete anything.
+ break;
+case FILE:
+ // If destination is a file,
+ if (srcType == Type.FILE) {
+// If source is a file, then delete the
[jira] [Work logged] (HIVE-25790) Make managed table copies handle updates (FileUtils)
[
https://issues.apache.org/jira/browse/HIVE-25790?focusedWorklogId=823023=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823023
]
ASF GitHub Bot logged work on HIVE-25790:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 10:35
Start Date: 03/Nov/22 10:35
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on code in PR #3582:
URL: https://github.com/apache/hive/pull/3582#discussion_r1012649913
##
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java:
##
@@ -492,20 +495,94 @@ private void doRegularCopyOnce(FileSystem sourceFs,
List srcList,
throw new IOException(e);
}
} else {
- //Destination should be empty
+ // Destination should have identical files only
if (overWrite) {
-deleteSubDirs(destinationFs, destination);
+leaveIdenticalFilesOnly(sourceFs, paths, destinationFs, destination);
}
copyFilesBetweenFS(sourceFs, paths, destinationFs, destination, false,
true);
}
}
- private void deleteSubDirs(FileSystem fs, Path path) throws IOException {
-//Delete the root path instead of doing a listing
-//This is more optimised
-delete(fs, path, true);
-//Recreate just the Root folder
-mkdirs(fs, path);
+ /**
+ * Leave identical files only between source and destination. It gets source
paths.
+ */
+ void leaveIdenticalFilesOnly(FileSystem sourceFs, Path[] srcPaths,
+ FileSystem destinationFs, Path destinationPath) throws IOException {
+for (Path srcPath : srcPaths) {
+ leaveIdenticalFilesOnlyFileByFile(sourceFs, srcPath,
+ destinationFs, new Path(destinationPath, srcPath.getName()));
+}
+ }
+
+ /**
+ * Leave identical files only between source and destination. It gets a
source path.
+ */
+ private void leaveIdenticalFilesOnlyFileByFile(FileSystem sourceFs, Path
srcPath,
+ FileSystem destinationFs, Path dstPath) throws IOException {
+boolean toDelete = false;
+Type dstType = getType(destinationFs, dstPath);
+Type srcType = getType(sourceFs, srcPath);
+LOG.debug("Source path: {}, type: {}", srcPath, srcType);
+LOG.debug("Destination path: {}, type: {}", dstPath, dstType);
+switch (dstType) {
+case NONE:
+ // If destination is none, then no need to delete anything.
+ break;
+case FILE:
+ // If destination is a file,
+ if (srcType == Type.FILE) {
+// If source is a file, then delete the destination file if it is not
same as source.
+toDelete = !FileUtils.isSameFile(sourceFs, srcPath, destinationFs,
dstPath);
+LOG.debug("Same file?: {}", !toDelete);
+ } else {
+// Otherwise, delete the destination file.
+toDelete = true;
+ }
+ break;
+case DIRECTORY:
+ // If destination is a directory,
+ if (srcType == Type.DIRECTORY) {
+// If both are directories, visit for children of both.
+Set bothChildNames = Stream.concat(
Review Comment:
why do you need concat? I think idea was to remove target-only files that
are not identical to the source. You should only compare the common set, isn't
it?
Issue Time Tracking
---
Worklog Id: (was: 823023)
Time Spent: 1h 10m (was: 1h)
> Make managed table copies handle updates (FileUtils)
>
>
> Key: HIVE-25790
> URL: https://issues.apache.org/jira/browse/HIVE-25790
> Project: Hive
> Issue Type: Sub-task
>Reporter: Haymant Mangla
>Assignee: Teddy Choi
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-26684) Upgrade maven-shade-plugin from 3.4.1 for bug fixes.
[ https://issues.apache.org/jira/browse/HIVE-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-26684. - Fix Version/s: 4.0.0 Resolution: Fixed > Upgrade maven-shade-plugin from 3.4.1 for bug fixes. > > > Key: HIVE-26684 > URL: https://issues.apache.org/jira/browse/HIVE-26684 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The Hive build currently runs with maven-shade-plugin version 3.1.1, released > in April 2018. This issue proposes to upgrade to the latest version, 3.4.1, > released in October 2022. See HIVE-26648 for an example of another patch that > is blocked due to a bug in version 3.1.1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26684) Upgrade maven-shade-plugin from 3.4.1 for bug fixes.
[ https://issues.apache.org/jira/browse/HIVE-26684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17628238#comment-17628238 ] Ayush Saxena commented on HIVE-26684: - Committed to master. Thanx [~cnauroth] for the contribution and [~ngangam] for the review!!! > Upgrade maven-shade-plugin from 3.4.1 for bug fixes. > > > Key: HIVE-26684 > URL: https://issues.apache.org/jira/browse/HIVE-26684 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The Hive build currently runs with maven-shade-plugin version 3.1.1, released > in April 2018. This issue proposes to upgrade to the latest version, 3.4.1, > released in October 2022. See HIVE-26648 for an example of another patch that > is blocked due to a bug in version 3.1.1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26684) Upgrade maven-shade-plugin from 3.4.1 for bug fixes.
[ https://issues.apache.org/jira/browse/HIVE-26684?focusedWorklogId=823018=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823018 ] ASF GitHub Bot logged work on HIVE-26684: - Author: ASF GitHub Bot Created on: 03/Nov/22 10:24 Start Date: 03/Nov/22 10:24 Worklog Time Spent: 10m Work Description: ayushtkn merged PR #3717: URL: https://github.com/apache/hive/pull/3717 Issue Time Tracking --- Worklog Id: (was: 823018) Time Spent: 1.5h (was: 1h 20m) > Upgrade maven-shade-plugin from 3.4.1 for bug fixes. > > > Key: HIVE-26684 > URL: https://issues.apache.org/jira/browse/HIVE-26684 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > The Hive build currently runs with maven-shade-plugin version 3.1.1, released > in April 2018. This issue proposes to upgrade to the latest version, 3.4.1, > released in October 2022. See HIVE-26648 for an example of another patch that > is blocked due to a bug in version 3.1.1. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26400) Provide docker images for Hive
[ https://issues.apache.org/jira/browse/HIVE-26400?focusedWorklogId=823005=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-823005 ] ASF GitHub Bot logged work on HIVE-26400: - Author: ASF GitHub Bot Created on: 03/Nov/22 09:42 Start Date: 03/Nov/22 09:42 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3448: URL: https://github.com/apache/hive/pull/3448#issuecomment-1301850083 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3448) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3448=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3448=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3448=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3448=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 823005) Time Spent: 3h 20m (was: 3h 10m) > Provide docker images for Hive > -- > > Key: HIVE-26400 > URL: https://issues.apache.org/jira/browse/HIVE-26400 > Project: Hive > Issue Type: Improvement > Components: Build Infrastructure >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Blocker > Labels: hive-4.0.0-must, pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > Make Apache Hive be able to run inside docker container in pseudo-distributed > mode, with MySQL/Derby as its back database, provide the following: > * Quick-start/Debugging/Prepare a test env for Hive; > * Tools to build target image with specified version of Hive and its > dependencies; > * Images can be used as the basis for the Kubernetes operator. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26697) Create vectorized Hive DeleteFilter for Iceberg tables
[
https://issues.apache.org/jira/browse/HIVE-26697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita reassigned HIVE-26697:
-
> Create vectorized Hive DeleteFilter for Iceberg tables
> --
>
> Key: HIVE-26697
> URL: https://issues.apache.org/jira/browse/HIVE-26697
> Project: Hive
> Issue Type: Sub-task
> Components: Iceberg integration
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
>
> Implement a DeleteFilter that changes an Iterable of VectorizedRowBatches in
> a way that whenever a new batch is seen in the {{next()}} method, we iterate
> its rows and apply the original Iceberg DeleteFilter on it. In VRBs we can
> utilize the {{selected}} array to mark rows as selected (not deleted), so
> essentially we're going to implement the {{markRowDeleted}} method.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26696) Implement row iterator for VectorizedRowBatch
[ https://issues.apache.org/jira/browse/HIVE-26696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita reassigned HIVE-26696: - > Implement row iterator for VectorizedRowBatch > - > > Key: HIVE-26696 > URL: https://issues.apache.org/jira/browse/HIVE-26696 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > > In order to utilize the existing DeleteFilter implementation of Iceberg we'll > need to provide a row iterator that iterates through the rows of a > VectorizedRowBatch instance. > This can be used during a read with positional deletes, but it will be > especially required for implementing equality deletes. > A HiveRow concept needs to be created along with a converter class that > provides row data information that's usable as an Iceberg Record. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26695) Vectorized reading of Iceberg V2 tables
[ https://issues.apache.org/jira/browse/HIVE-26695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita reassigned HIVE-26695: - > Vectorized reading of Iceberg V2 tables > --- > > Key: HIVE-26695 > URL: https://issues.apache.org/jira/browse/HIVE-26695 > Project: Hive > Issue Type: Improvement >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > > Currently HiveIcebergStorageHandler prevents proceeding to vectorized reading > code path for V2 Iceberg tables, as the feature for reading in delete > information and merging this to VectorizedRowBatches is missing. > This should serve as an umbrella jira to implement the feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26694) Populate file row position information during vectorized Iceberg reads
[ https://issues.apache.org/jira/browse/HIVE-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-26694: -- Parent: HIVE-26695 Issue Type: Sub-task (was: Improvement) > Populate file row position information during vectorized Iceberg reads > -- > > Key: HIVE-26694 > URL: https://issues.apache.org/jira/browse/HIVE-26694 > Project: Hive > Issue Type: Sub-task > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > > Currently we leverage VectorizedOrcRecordReader and > VectorizedParquetRecordReader during vectorized Iceberg reads. In order to > read V2 tables where delete files with positional deletes are present, we > need to make these record reader implementations provide the row number > information too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26695) Vectorized reading of Iceberg V2 tables
[ https://issues.apache.org/jira/browse/HIVE-26695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-26695: -- Component/s: Iceberg integration > Vectorized reading of Iceberg V2 tables > --- > > Key: HIVE-26695 > URL: https://issues.apache.org/jira/browse/HIVE-26695 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > > Currently HiveIcebergStorageHandler prevents proceeding to vectorized reading > code path for V2 Iceberg tables, as the feature for reading in delete > information and merging this to VectorizedRowBatches is missing. > This should serve as an umbrella jira to implement the feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-26694) Populate file row position information during vectorized Iceberg reads
[ https://issues.apache.org/jira/browse/HIVE-26694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita reassigned HIVE-26694: - > Populate file row position information during vectorized Iceberg reads > -- > > Key: HIVE-26694 > URL: https://issues.apache.org/jira/browse/HIVE-26694 > Project: Hive > Issue Type: Improvement > Components: Iceberg integration >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > > Currently we leverage VectorizedOrcRecordReader and > VectorizedParquetRecordReader during vectorized Iceberg reads. In order to > read V2 tables where delete files with positional deletes are present, we > need to make these record reader implementations provide the row number > information too. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (HIVE-26628) Iceberg table is created when running explain ctas command
[
https://issues.apache.org/jira/browse/HIVE-26628?focusedWorklogId=822992=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-822992
]
ASF GitHub Bot logged work on HIVE-26628:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 09:09
Start Date: 03/Nov/22 09:09
Worklog Time Spent: 10m
Work Description: sonarcloud[bot] commented on PR #3724:
URL: https://github.com/apache/hive/pull/3724#issuecomment-1301816827
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3724)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[0
Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[0
Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[0 Security
Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3724=false=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[0 Code
Smells](https://sonarcloud.io/project/issues?id=apache_hive=3724=false=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive=3724=coverage=list)
No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive=3724=duplicated_lines_density=list)
No Duplication information
Issue Time Tracking
---
Worklog Id: (was: 822992)
Time Spent: 1h 50m (was: 1h 40m)
> Iceberg table is created when running explain ctas command
> --
>
> Key: HIVE-26628
> URL: https://issues.apache.org/jira/browse/HIVE-26628
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> {code}
> create table source(a int, b string, c int);
> explain
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> create table tbl_ice stored by iceberg stored as orc tblproperties
> ('format-version'='2') as
> select a, b, c from source;
> {code}
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.parse.SemanticException: Table already exists:
> default.tbl_ice
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeCreateTable(SemanticAnalyzer.java:13963)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12528)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12693)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:460)
> at
>
[jira] [Work logged] (HIVE-26509) Introduce dynamic leader election in HMS
[ https://issues.apache.org/jira/browse/HIVE-26509?focusedWorklogId=822987=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-822987 ] ASF GitHub Bot logged work on HIVE-26509: - Author: ASF GitHub Bot Created on: 03/Nov/22 08:50 Start Date: 03/Nov/22 08:50 Worklog Time Spent: 10m Work Description: sonarcloud[bot] commented on PR #3567: URL: https://github.com/apache/hive/pull/3567#issuecomment-1301797258 Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive=3567) [](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=BUG) [10 Bugs](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=VULNERABILITY) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3567=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3567=false=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache_hive=3567=false=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=CODE_SMELL) [65 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive=3567=false=CODE_SMELL) [](https://sonarcloud.io/component_measures?id=apache_hive=3567=coverage=list) No Coverage information [](https://sonarcloud.io/component_measures?id=apache_hive=3567=duplicated_lines_density=list) No Duplication information Issue Time Tracking --- Worklog Id: (was: 822987) Time Spent: 6h (was: 5h 50m) > Introduce dynamic leader election in HMS > > > Key: HIVE-26509 > URL: https://issues.apache.org/jira/browse/HIVE-26509 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > Time Spent: 6h > Remaining Estimate: 0h > > From HIVE-21841 we have a leader HMS selected by configuring > metastore.housekeeping.leader.hostname on startup. This approach saves us > from running duplicated HMS's housekeeping tasks cluster-wide. > In this jira, we introduce another dynamic leader election: adopt hive lock > to implement the leader election. Once a HMS owns the lock, then it becomes > the leader, carries out the housekeeping tasks, and sends heartbeats to renew > the lock before timeout. If the leader fails to reclaim the lock, then stops > the already started tasks if it has, the electing event is audited. We can > achieve a more dynamic leader when the original goes down or in the public > cloud without well configured property, and reduce the leader’s burdens by > running these tasks among different leaders. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-13288) Confusing exception message in DagUtils.localizeResource
[
https://issues.apache.org/jira/browse/HIVE-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17628145#comment-17628145
]
László Bodor commented on HIVE-13288:
-
merged to master, thanks for fixing this [~cnauroth] and [~rbalamohan] for the
review!
> Confusing exception message in DagUtils.localizeResource
>
>
> Key: HIVE-13288
> URL: https://issues.apache.org/jira/browse/HIVE-13288
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Jeff Zhang
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I got the following exception when query through hive server2. And check the
> source code, it it due to some error when copying data from local to hdfs.
> But the IOException is ignored and assume that it is due to another thread is
> also writing. I don't think it make sense to assume that, at least should log
> the IOException.
> {code}
> LOG.info("Localizing resource because it does not exist: " + src + " to dest:
> " + dest);
> try {
> destFS.copyFromLocalFile(false, false, src, dest);
> } catch (IOException e) {
> LOG.info("Looks like another thread is writing the same file will
> wait.");
> int waitAttempts =
>
> conf.getInt(HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.varname,
>
> HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.defaultIntVal);
> long sleepInterval = HiveConf.getTimeVar(
> conf, HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL,
> TimeUnit.MILLISECONDS);
> LOG.info("Number of wait attempts: " + waitAttempts + ". Wait
> interval: "
> + sleepInterval);
> boolean found = false;
> {code}
> {noformat}
> 2016-03-15 11:25:39,921 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:getHiveJarDirectory(876)) - Jar dir is
> null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jeff/.hiveJars
> 2016-03-15 11:25:40,058 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(952)) - Localizing resource
> because it does not exist:
> file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to dest:
> hdfs://sandbox.hortonworks.com:8020/user/jeff/.hiveJars/hive-exec-1.2.1.2.3.2.0-2950-a97c953db414a4f792d868e2b0417578a61ccfa368048016926117b641b07f34.jar
> 2016-03-15 11:25:40,063 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(956)) - Looks like another
> thread is writing the same file will wait.
> 2016-03-15 11:25:40,064 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(963)) - Number of wait attempts:
> 5. Wait interval: 5000
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Shutting down
> the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Metastore
> shutdown complete.
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 2016-03-15 11:25:53,573 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/e43fbaab-a659-4331-90cb-0ea0b2098e25_resources
> 2016-03-15 11:25:53,577 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory: /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,582 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/hive/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,587 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory:
>
[jira] [Updated] (HIVE-13288) Confusing exception message in DagUtils.localizeResource
[
https://issues.apache.org/jira/browse/HIVE-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-13288:
Fix Version/s: 4.0.0-alpha-2
> Confusing exception message in DagUtils.localizeResource
>
>
> Key: HIVE-13288
> URL: https://issues.apache.org/jira/browse/HIVE-13288
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Jeff Zhang
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I got the following exception when query through hive server2. And check the
> source code, it it due to some error when copying data from local to hdfs.
> But the IOException is ignored and assume that it is due to another thread is
> also writing. I don't think it make sense to assume that, at least should log
> the IOException.
> {code}
> LOG.info("Localizing resource because it does not exist: " + src + " to dest:
> " + dest);
> try {
> destFS.copyFromLocalFile(false, false, src, dest);
> } catch (IOException e) {
> LOG.info("Looks like another thread is writing the same file will
> wait.");
> int waitAttempts =
>
> conf.getInt(HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.varname,
>
> HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.defaultIntVal);
> long sleepInterval = HiveConf.getTimeVar(
> conf, HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL,
> TimeUnit.MILLISECONDS);
> LOG.info("Number of wait attempts: " + waitAttempts + ". Wait
> interval: "
> + sleepInterval);
> boolean found = false;
> {code}
> {noformat}
> 2016-03-15 11:25:39,921 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:getHiveJarDirectory(876)) - Jar dir is
> null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jeff/.hiveJars
> 2016-03-15 11:25:40,058 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(952)) - Localizing resource
> because it does not exist:
> file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to dest:
> hdfs://sandbox.hortonworks.com:8020/user/jeff/.hiveJars/hive-exec-1.2.1.2.3.2.0-2950-a97c953db414a4f792d868e2b0417578a61ccfa368048016926117b641b07f34.jar
> 2016-03-15 11:25:40,063 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(956)) - Looks like another
> thread is writing the same file will wait.
> 2016-03-15 11:25:40,064 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(963)) - Number of wait attempts:
> 5. Wait interval: 5000
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Shutting down
> the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Metastore
> shutdown complete.
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 2016-03-15 11:25:53,573 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/e43fbaab-a659-4331-90cb-0ea0b2098e25_resources
> 2016-03-15 11:25:53,577 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory: /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,582 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/hive/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,587 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory:
> /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25/_tmp_space.db
> 2016-03-15 11:25:53,592 INFO
[jira] [Resolved] (HIVE-13288) Confusing exception message in DagUtils.localizeResource
[
https://issues.apache.org/jira/browse/HIVE-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor resolved HIVE-13288.
-
Resolution: Fixed
> Confusing exception message in DagUtils.localizeResource
>
>
> Key: HIVE-13288
> URL: https://issues.apache.org/jira/browse/HIVE-13288
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Jeff Zhang
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I got the following exception when query through hive server2. And check the
> source code, it it due to some error when copying data from local to hdfs.
> But the IOException is ignored and assume that it is due to another thread is
> also writing. I don't think it make sense to assume that, at least should log
> the IOException.
> {code}
> LOG.info("Localizing resource because it does not exist: " + src + " to dest:
> " + dest);
> try {
> destFS.copyFromLocalFile(false, false, src, dest);
> } catch (IOException e) {
> LOG.info("Looks like another thread is writing the same file will
> wait.");
> int waitAttempts =
>
> conf.getInt(HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.varname,
>
> HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.defaultIntVal);
> long sleepInterval = HiveConf.getTimeVar(
> conf, HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL,
> TimeUnit.MILLISECONDS);
> LOG.info("Number of wait attempts: " + waitAttempts + ". Wait
> interval: "
> + sleepInterval);
> boolean found = false;
> {code}
> {noformat}
> 2016-03-15 11:25:39,921 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:getHiveJarDirectory(876)) - Jar dir is
> null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jeff/.hiveJars
> 2016-03-15 11:25:40,058 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(952)) - Localizing resource
> because it does not exist:
> file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to dest:
> hdfs://sandbox.hortonworks.com:8020/user/jeff/.hiveJars/hive-exec-1.2.1.2.3.2.0-2950-a97c953db414a4f792d868e2b0417578a61ccfa368048016926117b641b07f34.jar
> 2016-03-15 11:25:40,063 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(956)) - Looks like another
> thread is writing the same file will wait.
> 2016-03-15 11:25:40,064 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(963)) - Number of wait attempts:
> 5. Wait interval: 5000
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Shutting down
> the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Metastore
> shutdown complete.
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 2016-03-15 11:25:53,573 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/e43fbaab-a659-4331-90cb-0ea0b2098e25_resources
> 2016-03-15 11:25:53,577 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory: /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,582 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/hive/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,587 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory:
> /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25/_tmp_space.db
> 2016-03-15 11:25:53,592 INFO
[jira] [Work logged] (HIVE-13288) Confusing exception message in DagUtils.localizeResource
[
https://issues.apache.org/jira/browse/HIVE-13288?focusedWorklogId=822954=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-822954
]
ASF GitHub Bot logged work on HIVE-13288:
-
Author: ASF GitHub Bot
Created on: 03/Nov/22 06:58
Start Date: 03/Nov/22 06:58
Worklog Time Spent: 10m
Work Description: abstractdog merged PR #3722:
URL: https://github.com/apache/hive/pull/3722
Issue Time Tracking
---
Worklog Id: (was: 822954)
Time Spent: 40m (was: 0.5h)
> Confusing exception message in DagUtils.localizeResource
>
>
> Key: HIVE-13288
> URL: https://issues.apache.org/jira/browse/HIVE-13288
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 1.2.1
>Reporter: Jeff Zhang
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> I got the following exception when query through hive server2. And check the
> source code, it it due to some error when copying data from local to hdfs.
> But the IOException is ignored and assume that it is due to another thread is
> also writing. I don't think it make sense to assume that, at least should log
> the IOException.
> {code}
> LOG.info("Localizing resource because it does not exist: " + src + " to dest:
> " + dest);
> try {
> destFS.copyFromLocalFile(false, false, src, dest);
> } catch (IOException e) {
> LOG.info("Looks like another thread is writing the same file will
> wait.");
> int waitAttempts =
>
> conf.getInt(HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.varname,
>
> HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_NUM_WAIT_ATTEMPTS.defaultIntVal);
> long sleepInterval = HiveConf.getTimeVar(
> conf, HiveConf.ConfVars.HIVE_LOCALIZE_RESOURCE_WAIT_INTERVAL,
> TimeUnit.MILLISECONDS);
> LOG.info("Number of wait attempts: " + waitAttempts + ". Wait
> interval: "
> + sleepInterval);
> boolean found = false;
> {code}
> {noformat}
> 2016-03-15 11:25:39,921 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:getHiveJarDirectory(876)) - Jar dir is
> null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/jeff/.hiveJars
> 2016-03-15 11:25:40,058 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(952)) - Localizing resource
> because it does not exist:
> file:/usr/hdp/2.3.2.0-2950/hive/lib/hive-exec-1.2.1.2.3.2.0-2950.jar to dest:
> hdfs://sandbox.hortonworks.com:8020/user/jeff/.hiveJars/hive-exec-1.2.1.2.3.2.0-2950-a97c953db414a4f792d868e2b0417578a61ccfa368048016926117b641b07f34.jar
> 2016-03-15 11:25:40,063 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(956)) - Looks like another
> thread is writing the same file will wait.
> 2016-03-15 11:25:40,064 INFO [HiveServer2-Background-Pool: Thread-249]:
> tez.DagUtils (DagUtils.java:localizeResource(963)) - Number of wait attempts:
> 5. Wait interval: 5000
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> thrift.ThriftCLIService (ThriftCLIService.java:OpenSession(294)) - Client
> protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-03-15 11:25:53,548 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Shutting down
> the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Shutting down the object store...
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> metastore.HiveMetaStore (HiveMetaStore.java:logInfo(747)) - 1: Metastore
> shutdown complete.
> 2016-03-15 11:25:53,549 INFO [HiveServer2-Handler-Pool: Thread-48]:
> HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(372)) -
> ugi=hive/sandbox.hortonworks@example.com ip=unknown-ip-addr
> cmd=Metastore shutdown complete.
> 2016-03-15 11:25:53,573 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created local
> directory: /tmp/e43fbaab-a659-4331-90cb-0ea0b2098e25_resources
> 2016-03-15 11:25:53,577 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641)) - Created HDFS
> directory: /tmp/hive/ambari-qa/e43fbaab-a659-4331-90cb-0ea0b2098e25
> 2016-03-15 11:25:53,582 INFO [HiveServer2-Handler-Pool: Thread-48]:
> session.SessionState (SessionState.java:createPath(641))
