[jira] [Updated] (HIVE-27688) hive MoveTask cannot show the correct exception message
[ https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27688: -- Labels: pull-request-available (was: ) > hive MoveTask cannot show the correct exception message > --- > > Key: HIVE-27688 > URL: https://issues.apache.org/jira/browse/HIVE-27688 > Project: Hive > Issue Type: Bug >Reporter: liang yu >Assignee: liang yu >Priority: Major > Labels: pull-request-available > Attachments: image-2023-09-13-17-39-27-864.png, > image-2023-09-13-17-40-02-981.png > > > I am using hive.version=3.1.3; hadoop.version=3.3.4. > Setting hive.load.data.owner to hive, and I used user ubd_by to execute sql. > When I try to insert *overwrite* to an {*}existing table partition{*}, I get > the exception that: Unable to move source /path/to/source to destination > /path/to/dest, which is a very regular error message, gives me no helpful > information. > > {code:java} > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source > hdfs://xl/user/ubd master/ubd_b_dwa. > db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination > hdfs://xl/user/ubd master/ubd_b_dwa. > db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code} > > But when I try to insert *into* an {*}existing table partition{*}, I get the > exception that: Unable to move source /path/to/source to destination > /path/to/dest as the file is not owned by hive and load data is also not ran > as hive. which gives me a very helpful error message that I should change the > hive.load.data.owner to hive. > > {code:java} > FAILED: Execution Error, return code l from org. > apache.hadoop.hive.ql.metadata.HiveException: org. > apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd > master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/ > hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-1 as > the file is not owned by hive and load data is also not ran as hive{code} > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27694) Include HiveIcebergSerDe in default list of serdes using HMS
[ https://issues.apache.org/jira/browse/HIVE-27694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam resolved HIVE-27694. -- Fix Version/s: 4.0.0 Resolution: Fixed Thank you for the review [~ayushtkn] and [~dengzh]. Fix has been merged to master. Closing the jira. > Include HiveIcebergSerDe in default list of serdes using HMS > > > Key: HIVE-27694 > URL: https://issues.apache.org/jira/browse/HIVE-27694 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > MetastoreConf has this default list of SerDes that use HMS to persist the > metadata. Iceberg tables also use have their metadata in HMS and hence can be > fetched from metastore. This SerDe needs to be added to this list as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27576) Backport of HIVE-21001: Upgrade to calcite-1.19
[ https://issues.apache.org/jira/browse/HIVE-27576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27576: -- Labels: pull-request-available (was: ) > Backport of HIVE-21001: Upgrade to calcite-1.19 > --- > > Key: HIVE-27576 > URL: https://issues.apache.org/jira/browse/HIVE-27576 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Kamal Sharma >Assignee: Kamal Sharma >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27698) Backport of HIVE-22398: Remove legacy code that can cause issue with new Yarn releases
[ https://issues.apache.org/jira/browse/HIVE-27698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-27698: -- Labels: pull-request-available (was: ) > Backport of HIVE-22398: Remove legacy code that can cause issue with new Yarn > releases > -- > > Key: HIVE-27698 > URL: https://issues.apache.org/jira/browse/HIVE-27698 > Project: Hive > Issue Type: Sub-task >Reporter: Aman Raj >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27698) Backport of HIVE-22398: Remove legacy code that can cause issue with new Yarn releases
Aman Raj created HIVE-27698: --- Summary: Backport of HIVE-22398: Remove legacy code that can cause issue with new Yarn releases Key: HIVE-27698 URL: https://issues.apache.org/jira/browse/HIVE-27698 Project: Hive Issue Type: Sub-task Reporter: Aman Raj -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27564) Add log for ZooKeeperTokenStore
[ https://issues.apache.org/jira/browse/HIVE-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766082#comment-17766082 ] Ayush Saxena commented on HIVE-27564: - Committed to master. Thanx [~1365976...@qq.com] for the contribution!!! > Add log for ZooKeeperTokenStore > --- > > Key: HIVE-27564 > URL: https://issues.apache.org/jira/browse/HIVE-27564 > Project: Hive > Issue Type: Improvement >Reporter: lvyanquan >Assignee: lvyanquan >Priority: Major > Labels: pull-request-available > Attachments: picture 1.png > > > when use zookeeper to store DelegationTokenIdentifier, we use > {code:java} > TokenStoreDelegationTokenSecretManager.encodeWritable(tokenIdentifier) {code} > to encode message of tokens and create the path in zookeeper, like > picture1.png > However, when this token was removed, error message will display > DelegationTokenIdentifier only, then we have no idea about whether this path > is still existed and when the path in zookeeper was created and deleted. > {code:java} > public byte[] retrievePassword(DelegationTokenIdentifier identifier) throws > InvalidToken { > DelegationTokenInformation info = this.tokenStore.getToken(identifier); > if (info == null) { > throw new InvalidToken("token expired or does not exist: " + > identifier); > } > // must reuse super as info.getPassword is not accessible > synchronized (this) { > try { > super.currentTokens.put(identifier, info); > return super.retrievePassword(identifier); > } finally { > super.currentTokens.remove(identifier); > } > } > } {code} > So, I try to add so log about the lifecycle of the tokenPath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27564) Add log for ZooKeeperTokenStore
[ https://issues.apache.org/jira/browse/HIVE-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-27564. - Fix Version/s: 4.0.0 Resolution: Fixed > Add log for ZooKeeperTokenStore > --- > > Key: HIVE-27564 > URL: https://issues.apache.org/jira/browse/HIVE-27564 > Project: Hive > Issue Type: Improvement >Reporter: lvyanquan >Assignee: lvyanquan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: picture 1.png > > > when use zookeeper to store DelegationTokenIdentifier, we use > {code:java} > TokenStoreDelegationTokenSecretManager.encodeWritable(tokenIdentifier) {code} > to encode message of tokens and create the path in zookeeper, like > picture1.png > However, when this token was removed, error message will display > DelegationTokenIdentifier only, then we have no idea about whether this path > is still existed and when the path in zookeeper was created and deleted. > {code:java} > public byte[] retrievePassword(DelegationTokenIdentifier identifier) throws > InvalidToken { > DelegationTokenInformation info = this.tokenStore.getToken(identifier); > if (info == null) { > throw new InvalidToken("token expired or does not exist: " + > identifier); > } > // must reuse super as info.getPassword is not accessible > synchronized (this) { > try { > super.currentTokens.put(identifier, info); > return super.retrievePassword(identifier); > } finally { > super.currentTokens.remove(identifier); > } > } > } {code} > So, I try to add so log about the lifecycle of the tokenPath. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-27697) Bump org.apache.commons:commons-compress from 1.23.0 to 1.24.0
Ayush Saxena created HIVE-27697: --- Summary: Bump org.apache.commons:commons-compress from 1.23.0 to 1.24.0 Key: HIVE-27697 URL: https://issues.apache.org/jira/browse/HIVE-27697 Project: Hive Issue Type: Improvement Reporter: Ayush Saxena PR from dependabot: https://github.com/apache/hive/pull/4701 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27664) AlterTableSetLocationAnalyzer threw a confusing exception "Cannot connect to namenode"
[ https://issues.apache.org/jira/browse/HIVE-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-27664. - Fix Version/s: 4.0.0 Resolution: Fixed > AlterTableSetLocationAnalyzer threw a confusing exception "Cannot connect to > namenode" > -- > > Key: HIVE-27664 > URL: https://issues.apache.org/jira/browse/HIVE-27664 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-beta-1 >Reporter: xiongyinke >Assignee: xiongyinke >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > > @Override > protected void analyzeCommand(TableName tableName, Map > partitionSpec, ASTNode command) > throws SemanticException { > String newLocation = unescapeSQLString(command.getChild(0).getText()); > try { > // To make sure host/port pair is valid, the status of the location does not > matter > FileSystem.get(new URI(newLocation), conf).getFileStatus(new > Path(newLocation)); > } catch (FileNotFoundException e) { > // Only check host/port pair is valid, whether the file exist or not does not > matter > } catch (Exception e) { > throw new SemanticException("Cannot connect to namenode, please check if > host/port pair for " + newLocation + > " is valid", e); > } > When the > "FileSystem.get(new URI(newLocation), conf).getFileStatus(new > Path(newLocation))" > code throws a "Permission denied" exception, the Beeline client will receive > the confusing exception "Cannot connect to namenode, please check if > host/port pair for". In reality, the issue is not with the namenode. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27664) AlterTableSetLocationAnalyzer threw a confusing exception "Cannot connect to namenode"
[ https://issues.apache.org/jira/browse/HIVE-27664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17766081#comment-17766081 ] Ayush Saxena commented on HIVE-27664: - Committed to master. Thanx [~bearbeer] for the contribution!!! > AlterTableSetLocationAnalyzer threw a confusing exception "Cannot connect to > namenode" > -- > > Key: HIVE-27664 > URL: https://issues.apache.org/jira/browse/HIVE-27664 > Project: Hive > Issue Type: Improvement >Affects Versions: 4.0.0-beta-1 >Reporter: xiongyinke >Assignee: xiongyinke >Priority: Major > Labels: pull-request-available > > @Override > protected void analyzeCommand(TableName tableName, Map > partitionSpec, ASTNode command) > throws SemanticException { > String newLocation = unescapeSQLString(command.getChild(0).getText()); > try { > // To make sure host/port pair is valid, the status of the location does not > matter > FileSystem.get(new URI(newLocation), conf).getFileStatus(new > Path(newLocation)); > } catch (FileNotFoundException e) { > // Only check host/port pair is valid, whether the file exist or not does not > matter > } catch (Exception e) { > throw new SemanticException("Cannot connect to namenode, please check if > host/port pair for " + newLocation + > " is valid", e); > } > When the > "FileSystem.get(new URI(newLocation), conf).getFileStatus(new > Path(newLocation))" > code throws a "Permission denied" exception, the Beeline client will receive > the confusing exception "Cannot connect to namenode, please check if > host/port pair for". In reality, the issue is not with the namenode. > -- This message was sent by Atlassian Jira (v8.20.10#820010)