Hi, I ran some tests regarding Spark's Delegation Token renewal mechanism. As I see, the concept here is simple: if I give my keytab file and client principal to Spark, it starts a token renewal thread, and renews the namenode delegation tokens after some time. This works fine.
Then I tried to run a long application (with HDFS operation in the end) without providing the keytab/principal to Spark, and I expected it to fail after the token expires. It turned out that this is not the case, the application finishes successfully without a delegation token renewal by Spark. My question is: how is that possible? Shouldn't a saveAsTextfile() fail after the namenode delegation token expired? Regards, Zsolt
