[ https://issues.apache.org/jira/browse/SPARK-28634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16900769#comment-16900769 ]
Yuming Wang commented on SPARK-28634: ------------------------------------- cc [~vanzin] I will provide more information if you need it. > Failed to start SparkSession with Keytab file > ---------------------------------------------- > > Key: SPARK-28634 > URL: https://issues.apache.org/jira/browse/SPARK-28634 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 3.0.0 > Reporter: Yuming Wang > Priority: Major > > {noformat} > [user-etl@hermesdevour002-700165 spark-3.0.0-SNAPSHOT-bin-2.7.4]$ > bin/spark-sql --master yarn --conf > spark.yarn.keytab=/apache/spark-2.3.0-bin-2.7.3/conf/user-etl.keytab --conf > spark.yarn.principal=user-...@prod.example.com > log4j:WARN No such property [maxFileSize] in > org.apache.log4j.rolling.RollingFileAppender. > log4j:WARN No such property [maxBackupIndex] in > org.apache.log4j.rolling.RollingFileAppender. > Exception in thread "main" org.apache.spark.SparkException: Application > application_1564558112805_1794 failed 2 times due to AM Container for > appattempt_1564558112805_1794_000002 exited with exitCode: 1 > For more detailed output, check the application tracking page: > https://0.0.0.0:8190/applicationhistory/app/application_1564558112805_1794 > Then click on links to logs of each attempt. > Diagnostics: Exception from container-launch. > Container id: container_e1987_1564558112805_1794_02_000001 > Exit code: 1 > Shell output: main : command provided 1 > main : run as user is user-etl > main : requested yarn user is user-etl > Getting exit code file... > Creating script paths... > Writing pid file... > Writing to tmp file > /hadoop/2/yarn/local/nmPrivate/application_1564558112805_1794/container_e1987_1564558112805_1794_02_000001/container_e1987_1564558112805_1794_02_000001.pid.tmp > Writing to cgroup task files... > Creating local dirs... > Launching container... > Getting exit code file... > Creating script paths... > Container exited with a non-zero exit code 1. Error file: prelaunch.err. > Last 4096 bytes of prelaunch.err : > Last 4096 bytes of stderr : > log4j:WARN No such property [maxFileSize] in > org.apache.log4j.rolling.RollingFileAppender. > log4j:WARN No such property [maxBackupIndex] in > org.apache.log4j.rolling.RollingFileAppender. > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/hadoop/2/yarn/local/usercache/user-etl/filecache/58/__spark_libs__4358879230136591830.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/apache/releases/hbase-1.1.2.2.6.4.1/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/apache/releases/hadoop-2.7.3.2.6.4.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Exception in thread "main" org.apache.spark.SparkException: Keytab file: > /apache/spark-2.3.0-bin-2.7.3/conf/user-etl.keytab does not exist > at > org.apache.spark.deploy.SparkHadoopUtil.loginUserFromKeytab(SparkHadoopUtil.scala:131) > at > org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:846) > at > org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:889) > at > org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) > Failing this attempt. Failing the application. > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:95) > at > org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62) > at > org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:185) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) > at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2466) > at > org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:948) > at scala.Option.getOrElse(Option.scala:138) > at > org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:939) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:48) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.<init>(SparkSQLCLIDriver.scala:315) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:166) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) > at > org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853) > at > org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:168) > at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:196) > at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:87) > at > org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:932) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:941) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > {noformat} > The keytab file is exist and can work in Spark 2.3 and 2.4. -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org