[
https://issues.apache.org/jira/browse/SPARK-21689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoukang updated SPARK-21689:
-----------------------------
Description:
When use yarn cluster mode,and we need scan hbase,there will be a case which
can not work:
If we put user jar on hdfs,when local classpath will has no hbase,which will
let get hbase token failed.Then later when job submitted to yarn, it will
failed since has no token to access hbase table.I mock three cases:
1：user jar is on classpath, and has hbase
{code:java}
17/08/10 13:48:03 INFO security.HadoopFSDelegationTokenProvider: Renewal
interval is 86400050 for token HDFS_DELEGATION_TOKEN
17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hive
17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hbase
17/08/10 13:48:05 INFO security.HBaseDelegationTokenProvider: Attempting to
fetch HBase security token.
{code}
Logs showing we can get token normally.
2：user jar on hdfs
{code:java}
17/08/10 13:43:58 WARN security.HBaseDelegationTokenProvider: Class
org.apache.hadoop.hbase.HBaseConfiguration not found.
17/08/10 13:43:58 INFO security.HBaseDelegationTokenProvider: Failed to get
token from service hbase
java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.security.token.TokenUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:41)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:112)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:109)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
{code}
Logs showing we can get token failed with ClassNotFoundException.
If we download user jar from remote first,then things will work correctly.
was:
When use yarn cluster mode,and we need scan hbase,there will be a case which
can not work:
If we put user jar on hdfs,when local classpath will has no hbase,which will
let get hbase token failed.Then later when job submitted to yarn, it will
failed since has no token to access hbase table.I mock three cases:
1：user jar is on classpath, and has hbase
{code:java}
17/08/10 13:48:03 INFO security.HadoopFSDelegationTokenProvider: Renewal
interval is 86400050 for token HDFS_DELEGATION_TOKEN
17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hive
17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hbase
17/08/10 13:48:05 INFO security.HBaseDelegationTokenProvider: Attempting to
fetch HBase security token.
{code}
2：user jar on hdfs
{code:java}
17/08/10 13:43:58 WARN security.HBaseDelegationTokenProvider: Class
org.apache.hadoop.hbase.HBaseConfiguration not found.
17/08/10 13:43:58 INFO security.HBaseDelegationTokenProvider: Failed to get
token from service hbase
java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.security.token.TokenUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:41)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:112)
at
org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:109)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
at
scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
{code}
If we download user jar from remote first,then things will work correctly.
> Spark submit will not get kerberos token token when hbase class not found
> -------------------------------------------------------------------------
>
> Key: SPARK-21689
> URL: https://issues.apache.org/jira/browse/SPARK-21689
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.1.0, 2.2.0
> Reporter: zhoukang
>
> When use yarn cluster mode,and we need scan hbase,there will be a case which
> can not work:
> If we put user jar on hdfs,when local classpath will has no hbase,which will
> let get hbase token failed.Then later when job submitted to yarn, it will
> failed since has no token to access hbase table.I mock three cases:
> 1：user jar is on classpath, and has hbase
> {code:java}
> 17/08/10 13:48:03 INFO security.HadoopFSDelegationTokenProvider: Renewal
> interval is 86400050 for token HDFS_DELEGATION_TOKEN
> 17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hive
> 17/08/10 13:48:03 INFO security.HadoopDelegationTokenManager: Service hbase
> 17/08/10 13:48:05 INFO security.HBaseDelegationTokenProvider: Attempting to
> fetch HBase security token.
> {code}
> Logs showing we can get token normally.
> 2：user jar on hdfs
> {code:java}
> 17/08/10 13:43:58 WARN security.HBaseDelegationTokenProvider: Class
> org.apache.hadoop.hbase.HBaseConfiguration not found.
> 17/08/10 13:43:58 INFO security.HBaseDelegationTokenProvider: Failed to get
> token from service hbase
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.security.token.TokenUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokens(HBaseDelegationTokenProvider.scala:41)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:112)
> at
> org.apache.spark.deploy.security.HadoopDelegationTokenManager$$anonfun$obtainDelegationTokens$2.apply(HadoopDelegationTokenManager.scala:109)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> at
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
> {code}
> Logs showing we can get token failed with ClassNotFoundException.
> If we download user jar from remote first,then things will work correctly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
