[
https://issues.apache.org/jira/browse/SPARK-35747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod KC updated SPARK-35747:
-----------------------------
Description:
In a secure Yarn cluster, even though HBase or Kafka, or Hive services are not
used in the user application, yarn client unnecessarily trying to generate
Delegations token from these services. This will add additional delays while
submitting spark application in a yarn cluster
Also during HBase delegation, token generation step in the application submit
stage, HBaseDelegationTokenProvider prints a full Exception Stack trace and it
causes a noisy warning.
{code:java}
WARN security.HBaseDelegationTokenProvider: Failed to get token from service
hbase
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
okenProvider.scala:93)
more than 100+ line exception stack trace{code}
Hence, if these services are not used in the user Application, it is better add
WARN message to disable Delegation Token generation for those services.
ie, spark.security.credentials.hbase.enabled=false ,
spark.security.credentials.hive.enabled=false ,
spark.security.credentials.kafka.enabled=false
was:
In a secure Yarn cluster where HBase service is down, even if the spark
application is not using HBase, during the application submit stage,
HBaseDelegationTokenProvider prints full Exception Stack trace and it causes a
noisy warning.
{code:java}
WARN security.HBaseDelegationTokenProvider: Failed to get token from service
hbase
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
okenProvider.scala:93)
more than 100 line exception stack trace{code}
Also, Application submission taking more time as
`HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn` retries the
connection to HBase master multiple times before it gives up. This slows down
the application submission steps. Hence, if HBase is not used in the user
Application, it is better to suggest user to disable HBase Delegation Token
generation.
ie, spark.security.credentials.hbase.enabled=false
> Avoid printing full Exception stack trace, if HBase/Kafka/Hive services are
> not running in a secure cluster
> ------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-35747
> URL: https://issues.apache.org/jira/browse/SPARK-35747
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.0.0, 3.1.0, 3.1.2
> Reporter: Vinod KC
> Priority: Minor
>
> In a secure Yarn cluster, even though HBase or Kafka, or Hive services are
> not used in the user application, yarn client unnecessarily trying to
> generate Delegations token from these services. This will add additional
> delays while submitting spark application in a yarn cluster
>
> Also during HBase delegation, token generation step in the application submit
> stage, HBaseDelegationTokenProvider prints a full Exception Stack trace and
> it causes a noisy warning.
> {code:java}
> WARN security.HBaseDelegationTokenProvider: Failed to get token from service
> hbase
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.spark.deploy.security.HBaseDelegationTokenProvider.obtainDelegationTokensWithHBaseConn(HBaseDelegationT
> okenProvider.scala:93)
> more than 100+ line exception stack trace{code}
> Hence, if these services are not used in the user Application, it is better
> add WARN message to disable Delegation Token generation for those services.
> ie, spark.security.credentials.hbase.enabled=false ,
> spark.security.credentials.hive.enabled=false ,
> spark.security.credentials.kafka.enabled=false
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
