Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Anyone know about this ? Or do I miss something here ? On Fri, Aug 7, 2015 at 4:20 PM, canan chen ccn...@gmail.com wrote: Is there any reason that historyserver use another property for the event log dir ? Thanks
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Hi Canan, The event log dir is a per-application setting whereas the history server is an independent service that serves history UIs from many applications. If you use history server locally then the `spark.history.fs.logDirectory` will happen to point to `spark.eventLog.dir`, but the use case it provides is broader than that. -Andrew 2015-08-19 5:13 GMT-07:00 canan chen ccn...@gmail.com: Anyone know about this ? Or do I miss something here ? On Fri, Aug 7, 2015 at 4:20 PM, canan chen ccn...@gmail.com wrote: Is there any reason that historyserver use another property for the event log dir ? Thanks
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Thanks Andrew, make sense. On Thu, Aug 20, 2015 at 4:40 AM, Andrew Or and...@databricks.com wrote: Hi Canan, The event log dir is a per-application setting whereas the history server is an independent service that serves history UIs from many applications. If you use history server locally then the `spark.history.fs.logDirectory` will happen to point to `spark.eventLog.dir`, but the use case it provides is broader than that. -Andrew 2015-08-19 5:13 GMT-07:00 canan chen ccn...@gmail.com: Anyone know about this ? Or do I miss something here ? On Fri, Aug 7, 2015 at 4:20 PM, canan chen ccn...@gmail.com wrote: Is there any reason that historyserver use another property for the event log dir ? Thanks
Re: Why use spark.history.fs.logDirectory instead of spark.eventLog.dir
Anyone know this ? Thanks On Fri, Aug 7, 2015 at 4:20 PM, canan chen ccn...@gmail.com wrote: Is there any reason that historyserver use another property for the event log dir ? Thanks
