You might also enable debug in: hadoop-env.sh
# Extra Java runtime options. Empty by default.
export HADOOP_OPTS=$HADOOP_OPTS -Djava.net.preferIPv4Stack=true
-Dsun.security.krb5.debug=true ${HADOOP_OPTS}”
and check that the principals are the same on the NameNode and DataNode.
and you can confirm the same on all nodes in hdfs-site.xml.
You can also ensure all nodes in the cluster are kerberized in core-site.xml
(no auth by default) :
property
namehadoop.security.authentication/name
valuekerberos/value
descriptionSet the authentication for the cluster. Valid values are:
simple or kerberos.
/description
/property
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html
Best Regards
Frank
On May 22, 2015, at 4:25 AM, Ted Yu yuzhih...@gmail.com wrote:
Can you share the exception(s) you encountered ?
Thanks
On May 22, 2015, at 12:33 AM, donhoff_h 165612...@qq.com wrote:
Hi,
My modified code is listed below, just add the SecurityUtil API. I don't
know which propertyKeys I should use, so I make 2 my own propertyKeys to
find the keytab and principal.
object TestHBaseRead2 {
def main(args: Array[String]) {
val conf = new SparkConf()
val sc = new SparkContext(conf)
val hbConf = HBaseConfiguration.create()
hbConf.set(dhao.keytab.file,//etc//spark//keytab//spark.user.keytab)
hbConf.set(dhao.user.principal,sp...@bgdt.dev.hrb)
SecurityUtil.login(hbConf,dhao.keytab.file,dhao.user.principal)
val conn = ConnectionFactory.createConnection(hbConf)
val tbl = conn.getTable(TableName.valueOf(spark_t01))
try {
val get = new Get(Bytes.toBytes(row01))
val res = tbl.get(get)
println(result:+res.toString)
}
finally {
tbl.close()
conn.close()
es.shutdown()
}
val rdd = sc.parallelize(Array(1,2,3,4,5,6,7,8,9,10))
val v = rdd.sum()
println(Value=+v)
sc.stop()
}
}
-- 原始邮件 --
发件人: yuzhihong;yuzhih...@gmail.com;
发送时间: 2015年5月22日(星期五) 下午3:25
收件人: donhoff_h165612...@qq.com;
抄送: Bill Qbill.q@gmail.com; useruser@spark.apache.org;
主题: Re: 回复： How to use spark to access HBase with Security enabled
Can you post the morning modified code ?
Thanks
On May 21, 2015, at 11:11 PM, donhoff_h 165612...@qq.com wrote:
Hi,
Thanks very much for the reply. I have tried the SecurityUtil. I can see
from log that this statement executed successfully, but I still can not
pass the authentication of HBase. And with more experiments, I found a new
interesting senario. If I run the program with yarn-client mode, the driver
can pass the authentication, but the executors can not. If I run the
program with yarn-cluster mode, both the driver and the executors can not
pass the authentication. Can anybody give me some clue with this info?
Many Thanks!
-- 原始邮件 --
发件人: yuzhihong;yuzhih...@gmail.com;
发送时间: 2015年5月22日(星期五) 凌晨5:29
收件人: donhoff_h165612...@qq.com;
抄送: Bill Qbill.q@gmail.com; useruser@spark.apache.org;
主题: Re: How to use spark to access HBase with Security enabled
Are the worker nodes colocated with HBase region servers ?
Were you running as hbase super user ?
You may need to login, using code similar to the following:
if (isSecurityEnabled()) {
SecurityUtil.login(conf, fileConfKey, principalConfKey, localhost);
}
SecurityUtil is hadoop class.
Cheers
On Thu, May 21, 2015 at 1:58 AM, donhoff_h 165612...@qq.com wrote:
Hi,
Many thanks for the help. My Spark version is 1.3.0 too and I run it on
Yarn. According to your advice I have changed the configuration. Now my
program can read the hbase-site.xml correctly. And it can also authenticate
with zookeeper successfully.
But I meet a new problem that is my program still can not pass the
authentication of HBase. Did you or anybody else ever meet such kind of
situation ? I used a keytab file to provide the principal. Since it can
pass the authentication of the Zookeeper, I am sure the keytab file is OK.
But it jsut can not pass the authentication of HBase. The exception is
listed below and could you or anybody else help me ? Still many many thanks!
Exception***
15/05/21 16:03:18 INFO zookeeper.ZooKeeper: Initiating client connection,
connectString=bgdt02.dev.hrb:2181,bgdt01.dev.hrb:2181,bgdt03.dev.hrb:2181
sessionTimeout=9 watcher=hconnection-0x4e142a710x0,
quorum=bgdt02.dev.hrb:2181,bgdt01.dev.hrb:2181,bgdt03.dev.hrb:2181,
baseZNode=/hbase
15/05/21 16:03:18 INFO zookeeper.Login: successfully logged in.
15/05/21 16:03:18 INFO zookeeper.Login: TGT refresh thread started.
15/05/21 16:03:18 INFO client.ZooKeeperSaslClient: Client will use GSSAPI
as SASL mechanism.
15/05/21 16:03:18 INFO zookeeper.ClientCnxn: Opening socket
