subject:"Re\: submitting spark job with kerberized Hadoop issue"

Re: submitting spark job with kerberized Hadoop issue

2016-08-21 Thread Aneela Saleem

Any update on this?

On Tuesday, 16 August 2016, Aneela Saleem  wrote:

> Thanks Steve,
>
> I went through this but still not able to fix the issue
>
> On Mon, Aug 15, 2016 at 2:01 AM, Steve Loughran  > wrote:
>
>> Hi,
>>
>> Just came across this while going through all emails I'd left unread over
>> my vacation.
>>
>> did you manage to fix this?
>>
>> 1. There's some notes I've taken on this topic:
>> https://www.gitbook.com/book/steveloughran/kerberos_and_hadoop/details
>>
>>  -look at "Error messages to fear" to see if this one has surfaced;
>> otherwise look at "low level secrets" to see how to start debugging things
>>
>>
>> On 5 Aug 2016, at 14:54, Aneela Saleem > > wrote:
>>
>> Hi all,
>>
>> I'm trying to connect to Kerberized Hadoop cluster using spark job. I
>> have kinit'd from command line. When i run the following job i.e.,
>>
>> *./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
>> spark/hadoop-master@platalyticsrealm --class
>> com.platalytics.example.spark.App --master spark://hadoop-master:7077
>> /home/vm6/project-1-jar-with-dependencies.jar
>> hdfs://hadoop-master:8020/text*
>>
>> I get the error:
>>
>> Caused by: java.io.IOException: 
>> org.apache.hadoop.security.AccessControlException:
>> Client cannot authenticate via:[TOKEN, KERBEROS]
>> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:415)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1628)
>>
>> Following are the contents of *spark-defaults.conf* file:
>>
>> spark.master spark://hadoop-master:7077
>> spark.eventLog.enabled   true
>> spark.eventLog.dir   hdfs://hadoop-master:8020/spark/logs
>> spark.serializer org.apache.spark.serializer.Kr
>> yoSerializer
>> spark.yarn.access.namenodes hdfs://hadoop-master:8020/
>> spark.yarn.security.tokens.hbase.enabled true
>> spark.yarn.security.tokens.hive.enabled true
>> spark.yarn.principal yarn/hadoop-master@platalyticsrealm
>> spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>>
>>
>> Also i have added following in *spark-env.sh* file:
>>
>> HOSTNAME=`hostname -f`
>> export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
>> -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
>> -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>>
>>
>> Please guide me, how to trace the issue?
>>
>> Thanks
>>
>>
>>
>

Re: submitting spark job with kerberized Hadoop issue

2016-08-16 Thread Aneela Saleem

Thanks Steve,

I went through this but still not able to fix the issue

On Mon, Aug 15, 2016 at 2:01 AM, Steve Loughran 
wrote:

> Hi,
>
> Just came across this while going through all emails I'd left unread over
> my vacation.
>
> did you manage to fix this?
>
> 1. There's some notes I've taken on this topic:
> https://www.gitbook.com/book/steveloughran/kerberos_and_hadoop/details
>
>  -look at "Error messages to fear" to see if this one has surfaced;
> otherwise look at "low level secrets" to see how to start debugging things
>
>
> On 5 Aug 2016, at 14:54, Aneela Saleem  wrote:
>
> Hi all,
>
> I'm trying to connect to Kerberized Hadoop cluster using spark job. I have
> kinit'd from command line. When i run the following job i.e.,
>
> *./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
> spark/hadoop-master@platalyticsrealm --class
> com.platalytics.example.spark.App --master spark://hadoop-master:7077
> /home/vm6/project-1-jar-with-dependencies.jar
> hdfs://hadoop-master:8020/text*
>
> I get the error:
>
> Caused by: java.io.IOException: 
> org.apache.hadoop.security.AccessControlException:
> Client cannot authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628)
>
> Following are the contents of *spark-defaults.conf* file:
>
> spark.master spark://hadoop-master:7077
> spark.eventLog.enabled   true
> spark.eventLog.dir   hdfs://hadoop-master:8020/spark/logs
> spark.serializer org.apache.spark.serializer.
> KryoSerializer
> spark.yarn.access.namenodes hdfs://hadoop-master:8020/
> spark.yarn.security.tokens.hbase.enabled true
> spark.yarn.security.tokens.hive.enabled true
> spark.yarn.principal yarn/hadoop-master@platalyticsrealm
> spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>
>
> Also i have added following in *spark-env.sh* file:
>
> HOSTNAME=`hostname -f`
> export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
> -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
> -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>
>
> Please guide me, how to trace the issue?
>
> Thanks
>
>
>

Re: submitting spark job with kerberized Hadoop issue

2016-08-08 Thread Aneela Saleem

Thanks Saisai and Ted,

I have already configured HBase security and it's working fine. I have also
done kinit before submitting job. Following is the code i'm trying to use


 System.setProperty("java.security.krb5.conf", "/etc/krb5.conf");
 System.setProperty("java.security.auth.login.config",
"/etc/hbase/conf/zk-jaas.conf");

  val hconf = HBaseConfiguration.create()
  val tableName = "emp"
  hconf.set("hbase.zookeeper.quorum", "hadoop-master")
  hconf.set(TableInputFormat.INPUT_TABLE, tableName)
  hconf.set("hbase.zookeeper.property.clientPort", "2181")
  hconf.set("hadoop.security.authentication", "kerberos")
  hconf.set("hbase.security.authentication", "kerberos")
  hconf.addResource(new Path("/etc/hbase/conf/core-site.xml"))
  hconf.addResource(new Path("/etc/hbase/conf/hbase-site.xml"))
  UserGroupInformation.setConfiguration(hconf)
  val keyTab = "/etc/hadoop/conf/spark.keytab"
  val ugi =
UserGroupInformation.loginUserFromKeytabAndReturnUGI("spark/hadoop-master@platalyticsrealm",
keyTab)
  UserGroupInformation.setLoginUser(ugi)
  ugi.doAs(new PrivilegedExceptionAction[Void]() {
   override def run(): Void = {
val conf = new SparkConf
val sc = new SparkContext(conf)
sc.addFile(keyTab)
var hBaseRDD = sc.newAPIHadoopRDD(hconf, classOf[TableInputFormat],
 classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],
 classOf[org.apache.hadoop.hbase.client.Result])
println("Number of Records found : " + hBaseRDD.count())
hBaseRDD.foreach(x => {
 println(new String(x._2.getRow()))
})
sc.stop()
return null
   }
  })

Please have a look.

Thanks

On Mon, Aug 8, 2016 at 9:34 AM, Ted Yu  wrote:

> The link in Jerry's response was quite old.
>
> Please see:
> http://hbase.apache.org/book.html#security
>
> Thanks
>
> On Sun, Aug 7, 2016 at 6:55 PM, Saisai Shao 
> wrote:
>
>> 1. Standalone mode doesn't support accessing kerberized Hadoop, simply
>> because it lacks the mechanism to distribute delegation tokens via cluster
>> manager.
>> 2. For the HBase token fetching failure, I think you have to do kinit to
>> generate tgt before start spark application (
>> http://hbase.apache.org/0.94/book/security.html).
>>
>> On Mon, Aug 8, 2016 at 12:05 AM, Aneela Saleem 
>> wrote:
>>
>>> Thanks Wojciech and Jacek!
>>>
>>> I tried with Spark on Yarn with kerberized cluster it works fine now.
>>> But now when i try to access Hbase through spark i get the following error:
>>>
>>> 2016-08-07 20:43:57,617 WARN  
>>> [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: 
>>> Exception encountered while connecting to the server : 
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
>>> GSSException: No valid credentials provided (Mechanism level: Failed to 
>>> find any Kerberos tgt)]
>>> 2016-08-07 20:43:57,619 ERROR 
>>> [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: 
>>> SASL authentication failed. The most likely cause is missing or invalid 
>>> credentials. Consider 'kinit'.
>>> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
>>> GSSException: No valid credentials provided (Mechanism level: Failed to 
>>> find any Kerberos tgt)]
>>> at 
>>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
>>> at 
>>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>> at 
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
>>> at 
>>> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
>>> at 
>>> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>>> at 
>>> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>>> at 
>>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
>>> at 
>>>

Re: submitting spark job with kerberized Hadoop issue

2016-08-07 Thread Ted Yu

The link in Jerry's response was quite old.

Please see:
http://hbase.apache.org/book.html#security

Thanks

On Sun, Aug 7, 2016 at 6:55 PM, Saisai Shao  wrote:

> 1. Standalone mode doesn't support accessing kerberized Hadoop, simply
> because it lacks the mechanism to distribute delegation tokens via cluster
> manager.
> 2. For the HBase token fetching failure, I think you have to do kinit to
> generate tgt before start spark application (http://hbase.apache.org/0.94/
> book/security.html).
>
> On Mon, Aug 8, 2016 at 12:05 AM, Aneela Saleem 
> wrote:
>
>> Thanks Wojciech and Jacek!
>>
>> I tried with Spark on Yarn with kerberized cluster it works fine now. But
>> now when i try to access Hbase through spark i get the following error:
>>
>> 2016-08-07 20:43:57,617 WARN  
>> [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: 
>> Exception encountered while connecting to the server : 
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
>> GSSException: No valid credentials provided (Mechanism level: Failed to find 
>> any Kerberos tgt)]
>> 2016-08-07 20:43:57,619 ERROR 
>> [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: SASL 
>> authentication failed. The most likely cause is missing or invalid 
>> credentials. Consider 'kinit'.
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
>> GSSException: No valid credentials provided (Mechanism level: Failed to find 
>> any Kerberos tgt)]
>>  at 
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
>>  at 
>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
>>  at java.security.AccessController.doPrivileged(Native Method)
>>  at javax.security.auth.Subject.doAs(Subject.java:415)
>>  at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
>>  at 
>> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
>>  at 
>> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>>  at 
>> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>>  at 
>> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
>>  at 
>> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
>>  at 
>> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
>>  at 
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
>>  at 
>> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
>>  at 
>> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
>>  at 
>> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
>>  at 
>> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>  at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>  at java.lang.Thread.run(Thread.java:745)
>> Caused by: GSSException: No valid credentials provided (Mechanism level: 
>> Failed to find any Kerberos tgt)
>>  at 
>> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>>  at 
>> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
>>  at 
>> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>>  at 
>> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
>>  at 
>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>>  at 
>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>>  at 
>>

Re: submitting spark job with kerberized Hadoop issue

2016-08-07 Thread Saisai Shao

1. Standalone mode doesn't support accessing kerberized Hadoop, simply
because it lacks the mechanism to distribute delegation tokens via cluster
manager.
2. For the HBase token fetching failure, I think you have to do kinit to
generate tgt before start spark application (
http://hbase.apache.org/0.94/book/security.html).

On Mon, Aug 8, 2016 at 12:05 AM, Aneela Saleem 
wrote:

> Thanks Wojciech and Jacek!
>
> I tried with Spark on Yarn with kerberized cluster it works fine now. But
> now when i try to access Hbase through spark i get the following error:
>
> 2016-08-07 20:43:57,617 WARN  
> [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: 
> Exception encountered while connecting to the server : 
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
> 2016-08-07 20:43:57,619 ERROR 
> [hconnection-0x24b5fa45-metaLookup-shared--pool2-t1] ipc.RpcClientImpl: SASL 
> authentication failed. The most likely cause is missing or invalid 
> credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by 
> GSSException: No valid credentials provided (Mechanism level: Failed to find 
> any Kerberos tgt)]
>   at 
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
>   at 
> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:415)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
>   at 
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
>   at 
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
>   at 
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
>   at 
> org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
>   at 
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
>   at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
>   at 
> org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: GSSException: No valid credentials provided (Mechanism level: 
> Failed to find any Kerberos tgt)
>   at 
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
>   at 
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
>   at 
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
>   at 
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
>   at 
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
>   at 
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
>   at 
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
>   ... 25 more
>
>

Re: submitting spark job with kerberized Hadoop issue

2016-08-07 Thread Aneela Saleem

Thanks Wojciech and Jacek!

I tried with Spark on Yarn with kerberized cluster it works fine now. But
now when i try to access Hbase through spark i get the following error:

2016-08-07 20:43:57,617 WARN
[hconnection-0x24b5fa45-metaLookup-shared--pool2-t1]
ipc.RpcClientImpl: Exception encountered while connecting to the
server : javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]
2016-08-07 20:43:57,619 ERROR
[hconnection-0x24b5fa45-metaLookup-shared--pool2-t1]
ipc.RpcClientImpl: SASL authentication failed. The most likely cause
is missing or invalid credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)]
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
at 
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
at 
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at 
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
at 
org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:201)
at 
org.apache.hadoop.hbase.client.ClientSmallScanner$SmallScannerCallable.call(ClientSmallScanner.java:180)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:360)
at 
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:334)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
at 
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)
at 
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at 
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
at 
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at 
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at 
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at 
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
... 25 more

Re: submitting spark job with kerberized Hadoop issue

2016-08-06 Thread Wojciech Pituła

What I can say, is that we successfully use spark on yarn with kerberized
cluster. One of my coworkers also tried using it in the same way as you
are(spark standalone with kerberized cluster) but as far as I remember, he
didn't succeed. I may be wrong, because I was not personally involved in
this use case, but I think that he concluded, that every executor of spark
standalone cluster must also be kinited.

pt., 5.08.2016 o 15:54 użytkownik Aneela Saleem 
napisał:

> Hi all,
>
> I'm trying to connect to Kerberized Hadoop cluster using spark job. I have
> kinit'd from command line. When i run the following job i.e.,
>
> *./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
> spark/hadoop-master@platalyticsrealm --class
> com.platalytics.example.spark.App --master spark://hadoop-master:7077
> /home/vm6/project-1-jar-with-dependencies.jar
> hdfs://hadoop-master:8020/text*
>
> I get the error:
>
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> Following are the contents of *spark-defaults.conf* file:
>
> spark.master spark://hadoop-master:7077
> spark.eventLog.enabled   true
> spark.eventLog.dir   hdfs://hadoop-master:8020/spark/logs
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.yarn.access.namenodes hdfs://hadoop-master:8020/
> spark.yarn.security.tokens.hbase.enabled true
> spark.yarn.security.tokens.hive.enabled true
> spark.yarn.principal yarn/hadoop-master@platalyticsrealm
> spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>
>
> Also i have added following in *spark-env.sh* file:
>
> HOSTNAME=`hostname -f`
> export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
> -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
> -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>
>
> Please guide me, how to trace the issue?
>
> Thanks
>
>

Re: submitting spark job with kerberized Hadoop issue

2016-08-06 Thread Jacek Laskowski

Hi Aneela,

I don't really know. I've never been using (or even toying with) Spark
Standalone to access a secured HDFS cluster. I however think the
settings won't work since they are for Spark on YARN (I would not be
surprised to know that it is not supported outside Spark on YARN).

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Sat, Aug 6, 2016 at 11:03 AM, Aneela Saleem  wrote:
> Hi Jacek!
>
> Thanks for your response. I am using spark standalone. I have secured hadoop
> cluster, Can you please guide me  wha to do if i want to access hadoop in my
> spark job?
>
> Thanks
>
> On Sat, Aug 6, 2016 at 12:34 AM, Jacek Laskowski  wrote:
>>
>> Just to make things clear...are you using Spark Standalone and Spark
>> on YARN-specific settings? I don't think it's gonna work.
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Fri, Aug 5, 2016 at 3:54 PM, Aneela Saleem 
>> wrote:
>> > Hi all,
>> >
>> > I'm trying to connect to Kerberized Hadoop cluster using spark job. I
>> > have
>> > kinit'd from command line. When i run the following job i.e.,
>> >
>> > ./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
>> > spark/hadoop-master@platalyticsrealm --class
>> > com.platalytics.example.spark.App --master spark://hadoop-master:7077
>> > /home/vm6/project-1-jar-with-dependencies.jar
>> > hdfs://hadoop-master:8020/text
>> >
>> > I get the error:
>> >
>> > Caused by: java.io.IOException:
>> > org.apache.hadoop.security.AccessControlException: Client cannot
>> > authenticate via:[TOKEN, KERBEROS]
>> > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:415)
>> > at
>> >
>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>> >
>> > Following are the contents of spark-defaults.conf file:
>> >
>> > spark.master spark://hadoop-master:7077
>> > spark.eventLog.enabled   true
>> > spark.eventLog.dir   hdfs://hadoop-master:8020/spark/logs
>> > spark.serializer
>> > org.apache.spark.serializer.KryoSerializer
>> > spark.yarn.access.namenodes hdfs://hadoop-master:8020/
>> > spark.yarn.security.tokens.hbase.enabled true
>> > spark.yarn.security.tokens.hive.enabled true
>> > spark.yarn.principal yarn/hadoop-master@platalyticsrealm
>> > spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>> >
>> >
>> > Also i have added following in spark-env.sh file:
>> >
>> > HOSTNAME=`hostname -f`
>> > export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
>> > -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
>> > -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>> >
>> >
>> > Please guide me, how to trace the issue?
>> >
>> > Thanks
>> >
>
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: submitting spark job with kerberized Hadoop issue

2016-08-06 Thread Aneela Saleem

Hi Jacek!

Thanks for your response. I am using spark standalone. I have secured
hadoop cluster, Can you please guide me  wha to do if i want to access
hadoop in my spark job?

Thanks

On Sat, Aug 6, 2016 at 12:34 AM, Jacek Laskowski  wrote:

> Just to make things clear...are you using Spark Standalone and Spark
> on YARN-specific settings? I don't think it's gonna work.
>
> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Fri, Aug 5, 2016 at 3:54 PM, Aneela Saleem 
> wrote:
> > Hi all,
> >
> > I'm trying to connect to Kerberized Hadoop cluster using spark job. I
> have
> > kinit'd from command line. When i run the following job i.e.,
> >
> > ./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
> > spark/hadoop-master@platalyticsrealm --class
> > com.platalytics.example.spark.App --master spark://hadoop-master:7077
> > /home/vm6/project-1-jar-with-dependencies.jar
> hdfs://hadoop-master:8020/text
> >
> > I get the error:
> >
> > Caused by: java.io.IOException:
> > org.apache.hadoop.security.AccessControlException: Client cannot
> > authenticate via:[TOKEN, KERBEROS]
> > at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:415)
> > at
> > org.apache.hadoop.security.UserGroupInformation.doAs(
> UserGroupInformation.java:1628)
> >
> > Following are the contents of spark-defaults.conf file:
> >
> > spark.master spark://hadoop-master:7077
> > spark.eventLog.enabled   true
> > spark.eventLog.dir   hdfs://hadoop-master:8020/spark/logs
> > spark.serializer org.apache.spark.serializer.
> KryoSerializer
> > spark.yarn.access.namenodes hdfs://hadoop-master:8020/
> > spark.yarn.security.tokens.hbase.enabled true
> > spark.yarn.security.tokens.hive.enabled true
> > spark.yarn.principal yarn/hadoop-master@platalyticsrealm
> > spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
> >
> >
> > Also i have added following in spark-env.sh file:
> >
> > HOSTNAME=`hostname -f`
> > export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
> > -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
> > -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
> >
> >
> > Please guide me, how to trace the issue?
> >
> > Thanks
> >
>

Re: submitting spark job with kerberized Hadoop issue

2016-08-05 Thread Jacek Laskowski

Just to make things clear...are you using Spark Standalone and Spark
on YARN-specific settings? I don't think it's gonna work.

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Fri, Aug 5, 2016 at 3:54 PM, Aneela Saleem  wrote:
> Hi all,
>
> I'm trying to connect to Kerberized Hadoop cluster using spark job. I have
> kinit'd from command line. When i run the following job i.e.,
>
> ./bin/spark-submit --keytab /etc/hadoop/conf/spark.keytab --principal
> spark/hadoop-master@platalyticsrealm --class
> com.platalytics.example.spark.App --master spark://hadoop-master:7077
> /home/vm6/project-1-jar-with-dependencies.jar hdfs://hadoop-master:8020/text
>
> I get the error:
>
> Caused by: java.io.IOException:
> org.apache.hadoop.security.AccessControlException: Client cannot
> authenticate via:[TOKEN, KERBEROS]
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
>
> Following are the contents of spark-defaults.conf file:
>
> spark.master spark://hadoop-master:7077
> spark.eventLog.enabled   true
> spark.eventLog.dir   hdfs://hadoop-master:8020/spark/logs
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.yarn.access.namenodes hdfs://hadoop-master:8020/
> spark.yarn.security.tokens.hbase.enabled true
> spark.yarn.security.tokens.hive.enabled true
> spark.yarn.principal yarn/hadoop-master@platalyticsrealm
> spark.yarn.keytab /etc/hadoop/conf/yarn.keytab
>
>
> Also i have added following in spark-env.sh file:
>
> HOSTNAME=`hostname -f`
> export SPARK_HISTORY_OPTS="-Dspark.history.kerberos.enabled=true
> -Dspark.history.kerberos.principal=spark/${HOSTNAME}@platalyticsrealm
> -Dspark.history.kerberos.keytab=/etc/hadoop/conf/spark.keytab"
>
>
> Please guide me, how to trace the issue?
>
> Thanks
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

Re: submitting spark job with kerberized Hadoop issue

10 matches

Site Navigation

Mail list logo

Footer information