[
https://issues.apache.org/jira/browse/HIVE-24342?focusedWorklogId=506712&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-506712
]
ASF GitHub Bot logged work on HIVE-24342:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 03/Nov/20 13:38
Start Date: 03/Nov/20 13:38
Worklog Time Spent: 10m
Work Description: umamaheswararao opened a new pull request #1639:
URL: https://github.com/apache/hive/pull/1639
https://issues.apache.org/jira/browse/HIVE-24342
### What changes were proposed in this pull request?
if the path is hdfs and if the fs is mounted fs, then we will use
resolvePath to recheck whether the path is really on hdfs. In the case ViewFS
based file systems, the target fs could be on some other fs, ex: o3fs://
### Why are the changes needed?
This is needed for supporting ViewHDFS like functionality as it supported
mounting other fs-es under hdfs paths.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
This has been tested in real cluster installation with mounting to o3fs
paths. The JIRA description provided the details about verification. After this
fix, querying from table succeeded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 506712)
Remaining Estimate: 0h
Time Spent: 10m
> isPathEncrypted should make sure resolved path also from HDFS
> -------------------------------------------------------------
>
> Key: HIVE-24342
> URL: https://issues.apache.org/jira/browse/HIVE-24342
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Shims
> Affects Versions: 3.1.2
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently isPathEncrypted will make sure path is from hdfs by check the path
> scheme is "hdfs"
> In the case if mounted ViewFileSystem based files systems like
> ViewFSOverloadScheme or ViewHDFS (HDFS-15289) may need o check resolved path
> is really hdfs.
> In ViewHDFS case, we can mount hdfs://ns1/test ---> o3fs://b.v.ozone1/test
> When user calling queries with the path hdfs://ns1/test, isPathEncrypted will
> think the path is from hdfs only as its checking path scheme.
>
> {code:java}
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> determine if hdfs://ns1/test is encrypted:
> java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
> DFS. Can't run on other fs:o3fs://bucket.volume.ozone1
> (state=42000,code=40000)
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> cd Closing: 0:
> jdbc:hive2://umag-1.umag.root.xxx.site:2181,umag-2.umag.root.xxx.site:2181,umag-5.umag.root.xxx.site:2181/default;password=root;principal=hive/umag-5.umag.root.xxx.s...@root.hwx.site;retries=5;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
> {code}
>
> So, here we should use resolvePath to make sure the resolved path really in
> hdfs. If the resolved path is not from hdfs (in above case, it o3fs path),
> then it will return false.
> After fixing this, the query is passing.:
>
> {code:java}
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
> INFO : Compiling
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
> select * from test30
> INFO : No Stats for default@test30, Columns: item, user_id, state, order_id
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:test30.order_id, type:bigint,
> comment:null), FieldSchema(name:test30.user_id, type:string, comment:null),
> FieldSchema(name:test30.item, type:string, comment:null),
> FieldSchema(name:test30.state, type:string, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb);
> Time taken: 4.47 seconds
> INFO : Executing
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
> select * from test30
> INFO : Completed executing
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb);
> Time taken: 0.09 seconds
> INFO : OK
> +------------------+-----------------+--------------+---------------+
> | test30.order_id | test30.user_id | test30.item | test30.state |
> +------------------+-----------------+--------------+---------------+
> | 1234 | u1 | iphone7 | CA |
> | 2345 | u1 | ipad | CA |
> | 3456 | u2 | desktop | NY |
>
>
> +------------------+-----------------+--------------+---------------+
> 11 rows selected (6.975 seconds)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
