SparkEventListener dropping events

We are seeing lots of stability problems with Spark 2.1.1 as a result of dropped events. We disabled the event log, which seemed to help, but many events are still being dropped, as in the example log below. I there any way for me to see what listener is backing up the queue? Is there any

Re: Bizarre UI Behavior after migration

, they don't get reported. > > I might be mistaken, if somebody has a good explanation, would also like > to hear. > > On Fri, May 19, 2017 at 5:45 PM, Miles Crawford <mil...@allenai.org> > wrote: > >> Hey ya'll, >> >> Trying to migrate from Spark 1.6.1 to

Re: Spark UI shows Jobs are processing, but the files are already written to S3

Could I be experiencing the same thing? https://www.dropbox.com/s/egtj1056qeudswj/sparkwut.png?dl=0 On Wed, Nov 16, 2016 at 10:37 AM, Shreya Agarwal wrote: > I think that is a bug. I have seen that a lot especially with long running > jobs where Spark skips a lot of

Bizarre UI Behavior after migration

Hey ya'll, Trying to migrate from Spark 1.6.1 to 2.1.0. I use EMR, and launched a new cluster using EMR 5.5, which runs spark 2.1.0. I updated my dependencies, and fixed a few API changes related to accumulators, and presto! my application was running on the new cluster. But the application UI

Bizarre behavior using Datasets/ML on Spark 2.0

Hello folks. I recently migrated my application to Spark 2.0, and everything worked well, except for one function that uses "toDS" and the ML libraries. This stage used to complete in 15 minutes or so on 1.6.2, and now takes almost two hours. The UI shows very strange behavior - completed stages

Re: History Server Refresh?

It is completed apps that are not showing up. I'm fine with incomplete apps not appearing. On Tue, Apr 12, 2016 at 6:43 AM, Steve Loughran <ste...@hortonworks.com> wrote: > > On 12 Apr 2016, at 00:21, Miles Crawford <mil...@allenai.org> wrote: > > Hey there. I have my s

History Server Refresh?

Hey there. I have my spark applications set up to write their event logs into S3 - this is super useful for ephemeral clusters, I can have persistent history even though my hosts go away. A history server is set up to view this s3 location, and that works fine too - at least on startup. The