[
https://issues.apache.org/jira/browse/HIVE-24342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17224193#comment-17224193
]
Uma Maheswara Rao G commented on HIVE-24342:
--------------------------------------------
Here is the full trace:
{code:java}
[e0a63527-ef51-40fa-9644-e08db4a310ce HiveServer2-Handler-Pool: Thread-3468]:
FAILED: SemanticException Unable to determine if hdfs://ns1/test is encrypted:
java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
DFS. Can't run on other fs:o3fs://bucket.volume.ozone1
org.apache.hadoop.hive.ql.parse.SemanticException: Unable to determine if
hdfs://ns1/test is encrypted: java.lang.UnsupportedOperationException: This
API:getEZForPath is specific to DFS. Can't run on other
fs:o3fs://bucket.volume.ozone1
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2078)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genResolvedParseTree(SemanticAnalyzer.java:12540)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12634)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:428)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:221)
at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:104)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:188)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:600)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:546)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:540)
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:127)
at
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:199)
at
org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:260)
at
org.apache.hive.service.cli.operation.Operation.run(Operation.java:274)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:565)
at
org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:551)
at
org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:567)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
at
org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:654)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
determine if hdfs://ns1/test is encrypted:
java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
DFS. Can't run on other fs:o3fs://bucket.volume.ozone1
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2447)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStrongestEncryptedTablePath(SemanticAnalyzer.java:2523)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getStagingDirectoryPathname(SemanticAnalyzer.java:2556)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2369)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:2070)
... 27 more
Caused by: java.lang.UnsupportedOperationException: This API:getEZForPath is
specific to DFS. Can't run on other fs:o3fs://bucket.volume.ozone1
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.checkDFS(ViewDistributedFileSystem.java:397)
at
org.apache.hadoop.hdfs.ViewDistributedFileSystem.getEZForPath(ViewDistributedFileSystem.java:1605)
at
org.apache.hadoop.hdfs.client.HdfsAdmin.getEncryptionZoneForPath(HdfsAdmin.java:375)
at
org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.getEncryptionZoneForPath(Hadoop23Shims.java:1238)
at
org.apache.hadoop.hive.shims.Hadoop23Shims$HdfsEncryptionShim.isPathEncrypted(Hadoop23Shims.java:1233)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.isPathEncrypted(SemanticAnalyzer.java:2442)
... 31 more{code}
> isPathEncrypted should make sure resolved path also from HDFS
> -------------------------------------------------------------
>
> Key: HIVE-24342
> URL: https://issues.apache.org/jira/browse/HIVE-24342
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Shims
> Affects Versions: 3.1.2
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
>
> Currently isPathEncrypted will make sure path is from hdfs by check the path
> scheme is "hdfs"
> In the case if mounted ViewFileSystem based files systems like
> ViewFSOverloadScheme or ViewHDFS (HDFS-15289) may need o check resolved path
> is really hdfs.
> In ViewHDFS case, we can mount hdfs://ns1/test ---> o3fs://b.v.ozone1/test
> When user calling queries with the path hdfs://ns1/test, isPathEncrypted will
> think the path is from hdfs only as its checking path scheme.
>
> {code:java}
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
> Error: Error while compiling statement: FAILED: SemanticException Unable to
> determine if hdfs://ns1/test is encrypted:
> java.lang.UnsupportedOperationException: This API:getEZForPath is specific to
> DFS. Can't run on other fs:o3fs://bucket.volume.ozone1
> (state=42000,code=40000)
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> cd Closing: 0:
> jdbc:hive2://umag-1.umag.root.xxx.site:2181,umag-2.umag.root.xxx.site:2181,umag-5.umag.root.xxx.site:2181/default;password=root;principal=hive/[email protected];retries=5;serviceDiscoveryMode=zooKeeper;user=root;zooKeeperNamespace=hiveserver2
> {code}
>
> So, here we should use resolvePath to make sure the resolved path really in
> hdfs. If the resolved path is not from hdfs (in above case, it o3fs path),
> then it will return false.
> After fixing this, the query is passing.:
>
> {code:java}
> 0: jdbc:hive2://umag-1.umag.root.xxx.site:218> select * from test30;
> INFO : Compiling
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
> select * from test30
> INFO : No Stats for default@test30, Columns: item, user_id, state, order_id
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema:
> Schema(fieldSchemas:[FieldSchema(name:test30.order_id, type:bigint,
> comment:null), FieldSchema(name:test30.user_id, type:string, comment:null),
> FieldSchema(name:test30.item, type:string, comment:null),
> FieldSchema(name:test30.state, type:string, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb);
> Time taken: 4.47 seconds
> INFO : Executing
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb):
> select * from test30
> INFO : Completed executing
> command(queryId=hive_20201031002253_1691548f-6fa8-4ea9-9cd4-87b70fe8f6bb);
> Time taken: 0.09 seconds
> INFO : OK
> +------------------+-----------------+--------------+---------------+
> | test30.order_id | test30.user_id | test30.item | test30.state |
> +------------------+-----------------+--------------+---------------+
> | 1234 | u1 | iphone7 | CA |
> | 2345 | u1 | ipad | CA |
> | 3456 | u2 | desktop | NY |
>
>
> +------------------+-----------------+--------------+---------------+
> 11 rows selected (6.975 seconds)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
