Github user steveloughran commented on a diff in the pull request: https://github.com/apache/spark/pull/9232#discussion_r43242989 --- Diff: yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala --- @@ -245,4 +247,55 @@ class YarnSparkHadoopUtilSuite extends SparkFunSuite with Matchers with Logging System.clearProperty("SPARK_YARN_MODE") } } + + test("Obtain tokens For HiveMetastore") { + val hadoopConf = new Configuration() + hadoopConf.set("hive.metastore.kerberos.principal", "bob") + // thrift picks up on port 0 and bails out, without trying to talk to endpoint + hadoopConf.set("hive.metastore.uris", "http://localhost:0") + val util = new YarnSparkHadoopUtil + val e = intercept[InvocationTargetException] { + util.obtainTokenForHiveMetastoreInner(hadoopConf, "alice") + } + assertNestedHiveException(e) + // expect exception trapping code to unwind this hive-side exception + assertNestedHiveException(intercept[InvocationTargetException] { + util.obtainTokenForHiveMetastore(hadoopConf) + }) + } + + def assertNestedHiveException(e: InvocationTargetException): Throwable = { + val inner = e.getCause + if (inner == null) { + fail("No inner cause", e) + } + if (!inner.isInstanceOf[HiveException]) { + fail(s"Not a hive exception", inner) + } + inner + } + + test("handleTokenIntrospectionFailure") { + val util = new YarnSparkHadoopUtil + // downgraded exceptions + util.handleTokenIntrospectionFailure("hive", new ClassNotFoundException("cnfe")) --- End diff -- As soon as this patch is in I'll turn to [SPARK-11317](https://issues.apache.org/jira/browse/SPARK-11317), which is essentially "apply the same catching, filtering and reporting strategy for HBase tokens as for Hive ones". It's not as critical as this one (token retrieval is working), but as nothing gets logged except "InvocationTargetException" with no stack trace, trying to recognise the issue is a Kerberos auth problem, let alone trying to fix it, is a weekend's effort, rather than 20 minutes worth. Because the policy goes in both places, having it separate and re-usable makes it a zero-cut-and-paste reuse, with that single test for failures without having to mock up failures across two separate clauses. And future maintenance costs are kept down if someone ever decides to change the policy again. Would you be happier if I cleaned up the HBase code as part of this same patch? Because I can and it will make the benefits of the factored out behaviour clearer. It's just messy to fix two things in one patch, especially if someone ever needs to play cherry pick or reverting games.
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org