Can you use the principal and keytab options in Spark submit? These should
circumvent this issue.
On 18 Nov 2016 1:01 p.m., "Abel Fernández" <mevsmys...@gmail.com> wrote:
> Hello,
>
> We are having problems with the delegation of the token in a secure
> cluster: Delegation Token can be issued only with kerberos or web
> authentication
>
> We have a spark process which is generating the hfiles to be loaded into
> hbase. To generate these hfiles, (we are using a back-ported version of the
> latest hbase/spark code), we are using this method HBaseRDDFunctions.
> hbaseBulkLoadThinRows.
>
> I think the problem is in the below piece of code. This function is
> executed in every partition of the rdd, when the executors are trying to
> execute the code, the executors do not have a valid kerberos credential and
> cannot execute anything.
>
> private def hbaseForeachPartition[T](configBroadcast:
> Broadcast[SerializableWritable[
> Configuration]],
> it: Iterator[T],
> f: (Iterator[T], Connection) =>
> Unit) = {
>
> val config = getConf(configBroadcast)
>
> applyCreds
> // specify that this is a proxy user
> val smartConn = HBaseConnectionCache.getConnection(config)
> f(it, smartConn.connection)
> smartConn.close()
> }
>
> I have attached the spark-submit and the complete error log trace. Has
> anyone faced this problem before?
>
> Thanks in advance.
>
> Regards,
> Abel.
> --
> Un saludo - Best Regards.
> Abel
>
