[ https://issues.apache.org/jira/browse/SPARK-25695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16644503#comment-16644503 ]
shahid commented on SPARK-25695: -------------------------------- Hi, This error seems, eventLog directory doesn't exist in your cluster. please create the directory and run. > Spark history server event log store problem > -------------------------------------------- > > Key: SPARK-25695 > URL: https://issues.apache.org/jira/browse/SPARK-25695 > Project: Spark > Issue Type: Bug > Components: Web UI > Affects Versions: 2.0.0 > Reporter: Si Chen > Priority: Major > > Envirment:spark 2.0.0 hadoop 2.7.3 > spark-default.conf > {code:java} > spark.driver.extraLibraryPath > /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 > spark.eventLog.dir file:/home/hdfs/event > spark.eventLog.enabled true > spark.executor.extraLibraryPath > /usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64 > spark.history.fs.logDirectory file:/home/hdfs/event > spark.history.kerberos.keytab none > spark.history.kerberos.principal none > spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider > spark.history.ui.port 18081 > spark.yarn.historyServer.address slave6.htdata.com:18081 > spark.yarn.queue default > {code} > I want to save eventLog to local disk. > When I submit spark job use client deploy mode, event log can write to local > disk. > But when I use cluster mode,The following problems arise.I am sure all > servers have this path. > {code:java} > 18/10/10 13:10:13 INFO cluster.SchedulerExtensionServices: Starting Yarn > extension services with app application_1538963194112_0033 and attemptId > Some(appattempt_1538963194112_0033_000001) > 18/10/10 13:10:13 INFO util.Utils: Successfully started service > 'org.apache.spark.network.netty.NettyBlockTransferService' on port 63016. > 18/10/10 13:10:13 INFO netty.NettyBlockTransferService: Server created on > 192.168.0.78:63016 > 18/10/10 13:10:13 INFO storage.BlockManagerMaster: Registering BlockManager > BlockManagerId(driver, 192.168.0.78, 63016) > 18/10/10 13:10:13 INFO storage.BlockManagerMasterEndpoint: Registering block > manager 192.168.0.78:63016 with 366.3 MB RAM, BlockManagerId(driver, > 192.168.0.78, 63016) > 18/10/10 13:10:13 INFO storage.BlockManagerMaster: Registered BlockManager > BlockManagerId(driver, 192.168.0.78, 63016) > 18/10/10 13:10:13 INFO handler.ContextHandler: Started > o.s.j.s.ServletContextHandler@32844dd3{/metrics/json,null,AVAILABLE} > 18/10/10 13:10:13 ERROR spark.SparkContext: Error initializing SparkContext. > java.io.FileNotFoundException: File file:/home/hdfs/event does not exist > at > org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:624) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:850) > at > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:614) > at > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:422) > at > org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:93) > at org.apache.spark.SparkContext.<init>(SparkContext.scala:516) > at > org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:836) > at > org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:84) > at > com.iiot.stream.spark.HTMonitorContext$.main(HTMonitorContext.scala:23) > at com.iiot.stream.spark.HTMonitorContext.main(HTMonitorContext.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627) > 18/10/10 13:10:13 INFO server.ServerConnector: Stopped > ServerConnector@4fd401cf{HTTP/1.1}{0.0.0.0:0} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@73fe17d4{/stages/stage/kill,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@11e5f9d4{/api,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5d6fa266{/,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5920363{/static,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@4494dda9{/executors/threadDump/json,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@5a32de89{/executors/threadDump,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@7c448c23{/executors/json,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@b0949d9{/executors,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@6a3e90c6{/environment/json,null,UNAVAILABLE} > 18/10/10 13:10:13 INFO handler.ContextHandler: Stopped > o.s.j.s.ServletContextHandler@6979e53f{/environment,null,UNAVAILABLE} > {code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org