You could probably instead of specifying --principal [principle] --keytab
[keytab] --proxy-user [proxied_user] ...
arguments just create/renew a kerberos ticket before submitting a job
$ kinit prinipal.name -kt keytab.file
$ spark-submit ...
Do you need impersonation / proxy user at all? I
After a bit more investigation, I found that it could be related to
impersonation on kerberized cluster.
Our job is started with the following command.
/usr/lib/spark/bin/spark-submit --master yarn-client --principal
[principle] --keytab [keytab] --proxy-user [proxied_user] ...
In application
2015-11-04 10:03:31,905 ERROR [Delegation Token Refresh Thread-0]
hdfs.KeyProviderCache (KeyProviderCache.java:createKeyProviderURI(87)) -
Could not find uri with key [dfs. encryption.key.provider.uri] to
create a keyProvider !!
Could it be related to HDFS-7931 ?
On Wed, Nov 4, 2015 at 12:30
We saw the following error happening in Spark Streaming job. Our job is
running on YARN with kerberos enabled.
First, warnings below were printed out, I only pasted a few but the
following was repeated hundred/thousand of times.
15/11/03 14:43:07 WARN UserGroupInformation:
