Are you caching a lot of RDD's? If so, maybe you should unpersist() the ones that you're not using. Also, if you're on 0.9, make sure spark.shuffle.spill is enabled (which it is by default). This allows your application to spill in-memory content to disk if necessary.
How much memory are you giving to your executors? The default, spark.executor.memory is 512m, which is quite low. Consider raising this. Checking the web UI is a good way to figure out your runtime memory usage. On Thu, Mar 27, 2014 at 9:22 AM, Ognen Duzlevski < og...@plainvanillagames.com> wrote: > Look at the tuning guide on Spark's webpage for strategies to cope with > this. > I have run into quite a few memory issues like these, some are resolved by > changing the StorageLevel strategy and employing things like Kryo, some are > solved by specifying the number of tasks to break down a given operation > into etc. > > Ognen > > > On 3/27/14, 10:21 AM, Sai Prasanna wrote: > > "java.lang.OutOfMemoryError: GC overhead limit exceeded" > > What is the problem. The same code, i run, one instance it runs in 8 > second, next time it takes really long time, say 300-500 seconds... > I see the logs a lot of GC overhead limit exceeded is seen. What should be > done ?? > > Please can someone throw some light on it ?? > > > > -- > *Sai Prasanna. AN* > *II M.Tech (CS), SSSIHL* > > > * Entire water in the ocean can never sink a ship, Unless it gets inside. > All the pressures of life can never hurt you, Unless you let them in.* > > >
