Re: Event Logging to HDFS on Standalone Cluster "In Progress"

2015-06-01 Thread Richard Marscher
Ah, apologies, I found an existing issue and fix has already gone out for this in 1.3.1 and up: https://issues.apache.org/jira/browse/SPARK-6036. On Mon, Jun 1, 2015 at 3:39 PM, Richard Marscher wrote: > It looks like it is possibly a race condition between removing the > IN_PROGRESS and buildin

Re: Event Logging to HDFS on Standalone Cluster "In Progress"

2015-06-01 Thread Richard Marscher
It looks like it is possibly a race condition between removing the IN_PROGRESS and building the history UI for the application. `AppClient` sends an `UnregisterApplication(appId)` message to the `Master` actor, which triggers the process to look for the app's eventLogs. If they are suffixed with `

Event Logging to HDFS on Standalone Cluster "In Progress"

2015-06-01 Thread Richard Marscher
Hi, In Spark 1.3.0 I've enabled event logging to write to an existing HDFS folder on a Standalone cluster. This is generally working, all the logs are being written. However, from the Master Web UI, the vast majority of completed applications are labeled as not having a history: http://xxx.xxx.xxx