Re: Spark UI shows finished when job had an error

Spark GUI runs by default on 4040 and if a job crashes (assuming you meant there was an issue with spark-submit), then the GUI will disconnect. GUI is not there for diagnostics as it reports on statistics. My inclination would be to look at the YARN log files assuming you are using YARN as your

Re: Spark UI shows finished when job had an error

Hi, Can you please see the query plan (in case you are using a query)? There is a very high chance that the query was broken into multiple steps and only a subsequent step failed. Regards, Gourav Sengupta On Fri, Jun 17, 2016 at 2:49 PM, Sumona Routh wrote: > Hi there, >

Re: Spark UI shows finished when job had an error

Hi, How do you access Cassandra? Could that connector not have sent a SparkListenerEvent to inform about failure? Jacek On 17 Jun 2016 3:50 p.m., "Sumona Routh" wrote: > Hi there, > Our Spark job had an error (specifically the Cassandra table definition > did not match what

Spark UI shows finished when job had an error

Hi there, Our Spark job had an error (specifically the Cassandra table definition did not match what was in Cassandra), which threw an exception that logged out to our spark-submit log. However ,the UI never showed any failed stage or job. It appeared as if the job finished without error, which is