[jira] [Commented] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848656#comment-17848656
]
Raghav Aggarwal commented on TEZ-4557:
--
[~ayushtkn] , can you provide your inputs here?
I don't have hive4 cluster(with ranger) to test this but irrespective of that,
I think the issue will come. My understanding is, hadoop depends on httpclient
jar and it was shipping it with hadoop-common transitively but after exclusion
this hadoop functionality is broken unless tez had direct dependency on
httpclient, which is not the case.
Exclusion from tez would have made sense if there were 2 different version of
httpclient (one coming transitively from hadoop and other from tez via direct
dependency).
> Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
> ---
>
> Key: TEZ-4557
> URL: https://issues.apache.org/jira/browse/TEZ-4557
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When insert data into table located in encryption zone using Hive with tez
> fails as the httpclient jar has been excluded from hadoop transitive
> dependency. Same query passes with MR.
> Tez: 0.10.2,0.10.3
> Hadoop: 3.3.6
> Hive: 3.1.2
>
> Steps to reproduce issue:
> 1. Create a encryption key using ranger keyadmin user.
> 2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
> 3. create table tbl(id int) location '/user/raghav/encrypt_zone';
> 4. insert into tbl values(1);
>
> Stacktrace:
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/http/client/utils/URIBuilder
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at
[jira] [Comment Edited] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842941#comment-17842941
]
Raghav Aggarwal edited comment on TEZ-4557 at 5/2/24 11:38 AM:
---
[~ayushtkn], I am using Hive 3.1.2 , hadoop 3.3.6 and tez 0.10.2.
The issue should happen in hive 4 with tez 0.10.3, as httpclient jar is missing
from tez/lib. Haven't tested it explicitly with those version as ranger
integration will be required.
was (Author: JIRAUSER295901):
I am using Hive 3.1.2 , hadoop 3.3.6 and tez 0.10.2.
The issue should happen in hive 4 with tez 0.10.3, as httpclient jar is missing
from tez/lib.
> Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
> ---
>
> Key: TEZ-4557
> URL: https://issues.apache.org/jira/browse/TEZ-4557
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
>
> When insert data into table located in encryption zone using Hive with tez
> fails as the httpclient jar has been excluded from hadoop transitive
> dependency. Same query passes with MR.
> Tez: 0.10.2,0.10.3
> Hadoop: 3.3.6
> Hive: 3.1.2
>
> Steps to reproduce issue:
> 1. Create a encryption key using ranger keyadmin user.
> 2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
> 3. create table tbl(id int) location '/user/raghav/encrypt_zone';
> 4. insert into tbl values(1);
>
> Stacktrace:
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/http/client/utils/URIBuilder
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
>
[jira] [Commented] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842941#comment-17842941
]
Raghav Aggarwal commented on TEZ-4557:
--
I am using Hive 3.1.2 , hadoop 3.3.6 and tez 0.10.2.
The issue should happen in hive 4 with tez 0.10.3, as httpclient jar is missing
from tez/lib.
> Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
> ---
>
> Key: TEZ-4557
> URL: https://issues.apache.org/jira/browse/TEZ-4557
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
>
> When insert data into table located in encryption zone using Hive with tez
> fails as the httpclient jar has been excluded from hadoop transitive
> dependency. Same query passes with MR.
> Tez: 0.10.2,0.10.3
> Hadoop: 3.3.6
> Hive: 3.1.2
>
> Steps to reproduce issue:
> 1. Create a encryption key using ranger keyadmin user.
> 2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
> 3. create table tbl(id int) location '/user/raghav/encrypt_zone';
> 4. insert into tbl values(1);
>
> Stacktrace:
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/http/client/utils/URIBuilder
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
>
[jira] [Updated] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated TEZ-4557:
-
Description:
When insert data into table located in encryption zone using Hive with tez
fails as the httpclient jar has been excluded from hadoop transitive
dependency. Same query passes with MR.
Tez: 0.10.2,0.10.3
Hadoop: 3.3.6
Hive: 3.1.2
Steps to reproduce issue:
1. Create a encryption key using ranger keyadmin user.
2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
3. create table tbl(id int) location '/user/raghav/encrypt_zone';
4. insert into tbl values(1);
Stacktrace:
{code:java}
Caused by: java.lang.NoClassDefFoundError:
org/apache/http/client/utils/URIBuilder
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
at
org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
at
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
[jira] [Commented] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17842937#comment-17842937
]
Raghav Aggarwal commented on TEZ-4557:
--
CC [~abstractdog]
> Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
> ---
>
> Key: TEZ-4557
> URL: https://issues.apache.org/jira/browse/TEZ-4557
> Project: Apache Tez
> Issue Type: Bug
>Reporter: Raghav Aggarwal
>Assignee: Raghav Aggarwal
>Priority: Major
>
> When insert data into table located in encryption zone using Hive with tez
> fails as the httpclient jar has been excluded from hadoop transitive
> dependency.
> Tez: 0.10.2,0.10.3
> Hadoop: 3.3.6
> Hive: 3.1.2
>
> Steps to reproduce issue:
> 1. Create a encryption key using ranger keyadmin user.
> 2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
> 3. create table tbl(id int) location '/user/raghav/encrypt_zone';
> 4. insert into tbl values(1);
>
> Stacktrace:
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/http/client/utils/URIBuilder
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
> at
> org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
> at
> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
> at
> org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
> at
> org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
> at
> org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
> at
[jira] [Updated] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated TEZ-4557:
-
Description:
When insert data into table located in encryption zone using Hive with tez
fails as the httpclient jar has been excluded from hadoop transitive
dependency.
Tez: 0.10.2,0.10.3
Hadoop: 3.3.6
Hive: 3.1.2
Steps to reproduce issue:
1. Create a encryption key using ranger keyadmin user.
2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
3. create table tbl(id int) location '/user/raghav/encrypt_zone';
4. insert into tbl values(1);
Stacktrace:
{code:java}
Caused by: java.lang.NoClassDefFoundError:
org/apache/http/client/utils/URIBuilder
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
at
org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
at
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
at
[jira] [Updated] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
[
https://issues.apache.org/jira/browse/TEZ-4557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Raghav Aggarwal updated TEZ-4557:
-
Description:
When insert data into table located in encryption zone using Hive with tez
fails as the httpclient jar has been excluded from hadoop transitive
dependency.
Steps to reproduce issue:
1. Create a encryption key using ranger keyadmin user.
2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
3. create table tbl(id int) location '/user/raghav/encrypt_zone';
4. insert into tbl values(1);
Stacktrace:
{code:java}
Caused by: java.lang.NoClassDefFoundError:
org/apache/http/client/utils/URIBuilder
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
at
org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
at
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:556)
[jira] [Created] (TEZ-4557) Revert TEZ-4303, NoClassDefFoundError because of missing httpclient jar
Raghav Aggarwal created TEZ-4557:
Summary: Revert TEZ-4303, NoClassDefFoundError because of missing
httpclient jar
Key: TEZ-4557
URL: https://issues.apache.org/jira/browse/TEZ-4557
Project: Apache Tez
Issue Type: Bug
Reporter: Raghav Aggarwal
Assignee: Raghav Aggarwal
Steps to reproduce issue:
1. Create a encryption key using ranger keyadmin user.
2. hdfs crypto -createZone -keyName test_key -path /user/raghav/encrypt_zone
3. create table tbl(id int) location '/user/raghav/encrypt_zone';
4. insert into tbl values(1);
Stacktrace:
{code:java}
Caused by: java.lang.NoClassDefFoundError:
org/apache/http/client/utils/URIBuilder
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.createURL(KMSClientProvider.java:468)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:823)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:354)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:175)
at
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:350)
at
org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:535)
at
org.apache.hadoop.hdfs.HdfsKMSUtil.decryptEncryptedDataEncryptionKey(HdfsKMSUtil.java:216)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:1002)
at
org.apache.hadoop.hdfs.DFSClient.createWrappedOutputStream(DFSClient.java:983)
at
org.apache.hadoop.hdfs.DistributedFileSystem.safelyCreateWrappedOutputStream(DistributedFileSystem.java:734)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$300(DistributedFileSystem.java:149)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:572)
at
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:566)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:580)
at
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:507)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1109)
at
org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:81)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:297)
at
org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:282)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:801)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:752)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:922)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.forwardUDTFOutput(UDTFOperator.java:133)
at
org.apache.hadoop.hive.ql.udf.generic.UDTFCollector.collect(UDTFCollector.java:45)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.forward(GenericUDTF.java:110)
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDTFInline.process(GenericUDTFInline.java:64)
at
org.apache.hadoop.hive.ql.exec.UDTFOperator.process(UDTFOperator.java:116)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:926)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
at org.apache.hadoop.hive.ql.exec.Operator.baseForward(Operator.java:993)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:939)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:125)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:154)
at
[jira] [Created] (TEZ-4540) Reading proto data more than 2GB from multiple splits fails
Raghav Aggarwal created TEZ-4540: Summary: Reading proto data more than 2GB from multiple splits fails Key: TEZ-4540 URL: https://issues.apache.org/jira/browse/TEZ-4540 Project: Apache Tez Issue Type: Bug Affects Versions: 0.10.2 Reporter: Raghav Aggarwal Refer to this: HIVE-28026 and https://github.com/apache/hive/pull/5033 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (TEZ-3392) Could not find or load main class org.apache.tez.dag.app.DAGAppMaster
[ https://issues.apache.org/jira/browse/TEZ-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17632384#comment-17632384 ] Raghav Aggarwal commented on TEZ-3392: -- I am also facing this issue. Can anyone look into this? > Could not find or load main class org.apache.tez.dag.app.DAGAppMaster > - > > Key: TEZ-3392 > URL: https://issues.apache.org/jira/browse/TEZ-3392 > Project: Apache Tez > Issue Type: Bug >Affects Versions: 0.8.4 > Environment: CentOS release 6.8 (Final) x64bit > Hadoop 2.7.2 > apache tez 0.8.4 >Reporter: amethystic >Priority: Major > > Running Tez examples but ResourceManager showed it failed due to: > Could not find or load main class org.apache.tez.dag.app.DAGAppMaster > Sounds like there is an already-prebuilt .tar.gz > file(apache-tez-0.8.4-bin.tar.gz) staring 0.8.4, so I used this file directly > instead of compiling by myself and put it onto HDFS path: /apps/tez > My tez-site.xml is: > > > > > tez.lib.uris > > hdfs://localhost:8500/apps/tez/apache-tez-0.8.4-bin.tar.gz > > > mapreduce.framework.name was also set to yarn-tez in mapreduce-site.xml. > All env variables were set in hadoop-env.sh: > export TEZ_JARS=/mnt/disk/huxi/tez > export TEZ_CONF_DIR=/mnt/disk/huxi/hadoop/etc/hadoop > export > HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/mnt/disk/huxi/tez/*:/mnt/disk/huxi/tez/lib/*:/mnt/disk/huxi/tez/conf/* > When issuing command "bin/hadoop jar > /mnt/disk/huxi/tez/tez-examples-0.8.4.jar orderedwordcount ./NOTICE.txt > /out", RM showed the task was failed: > > Application application_1470107708886_0001 failed 2 times due to AM Container > for appattempt_1470107708886_0001_02 exited with exitCode: 1 > For more detailed output, check application tracking > page:http://iZ25f2qedc7Z:8088/cluster/app/application_1470107708886_0001Then, > click on links to logs of each attempt. > Diagnostics: Exception from container-launch. > Container id: container_1470107708886_0001_02_01 > Exit code: 1 > Stack trace: ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) > at org.apache.hadoop.util.Shell.run(Shell.java:456) > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 1 > Failing this attempt. Failing the application. > After checking the log, stederr complained that : > Error: Could not find or load main class org.apache.tez.dag.app.DAGAppMaster > Weird enough, tez-dag-0.8.4.jar should be in the classpath. So please be > advice. -- This message was sent by Atlassian Jira (v8.20.10#820010)
