(-dev, +user. dev is for Spark development, not for questions about using Spark.)
You haven't posted code here or the actual error. But you might be running into SPARK-15754. Or into other issues with yarn-client mode and "--principal / --keytab" (those have known issues in client mode). If you have the above fix, you should be able to run the SparkContext in client mode inside a UGI.doAs() block, after you login the user, and later stop the context and start a new one. (And don't use "--principal" / "--keytab" in that case.) On Thu, Dec 15, 2016 at 1:46 PM, Alexey Klimov <kli...@megaputer.ru> wrote: > Hello, my question is the continuation of a problem I described here > <http://stackoverflow.com/questions/40834482/delegation-token-can-be-issued-only-with-kerberos-or-web-authentication-when-r> > . > > I've done some investigation and found out that nameNode.getDelegationToken > is called during constructing SparkContext even if delegation token is > already presented in token list of current logged user in object of > UserGroupInforation class. The problem doesn't occur when waiting time > before constructing a new context is less than 10 seconds, because rpc > connection to namenode just isn't resetting because of > ipc.client.connection.maxidletime property. > > As a workaround of this problem I do login from keytab before every > constructing of SparkContext, which basically just resets token list of > current logged user (as well as whole user structure) and the problem seems > to be gone. Still I'm not really sure that it is correct way to deal with > SparkContext. > > Having found a reason of the problem, I've got 2 assumptions now: > First - SparkContext was designed to be restarted during JVM run and > behaviour above is just a bug. > Second - it wasn't and I'm just using SparkContext in a wrong manner. > > Since I haven't found any related bug in Jira and any solution on the > internet (as well as too many users facing this error) I tend to think that > it is rather a not allowed usage of SparkContext. > > Is that correct? > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Is-restarting-of-SparkContext-allowed-tp20240.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
