Re: ExternalAppendOnlyMap: Spilling in-memory map

2014-05-22 Thread Andrew Ash
ache.org/docs/latest/tuning.html#data-serialization >> >> >> On Wed, May 21, 2014 at 2:35 PM, Mohit Jaggi wrote: >> >>> Hi, >>> >>> I changed my application to use Joda time instead of java.util.Date and >>> I started getting this: >>

Re: ExternalAppendOnlyMap: Spilling in-memory map

2014-05-22 Thread Mohit Jaggi
Kryo? > > http://spark.apache.org/docs/latest/tuning.html#data-serialization > > > On Wed, May 21, 2014 at 2:35 PM, Mohit Jaggi wrote: > >> Hi, >> >> I changed my application to use Joda time instead of java.util.Date and I >> started getting this: >&g

Re: ExternalAppendOnlyMap: Spilling in-memory map

2014-05-21 Thread Andrew Ash
Time is registered with Kryo? http://spark.apache.org/docs/latest/tuning.html#data-serialization On Wed, May 21, 2014 at 2:35 PM, Mohit Jaggi wrote: > Hi, > > I changed my application to use Joda time instead of java.util.Date and I > started getting this: > > WARN ExternalAppendOnlyMa

ExternalAppendOnlyMap: Spilling in-memory map

2014-05-21 Thread Mohit Jaggi
Hi, I changed my application to use Joda time instead of java.util.Date and I started getting this: WARN ExternalAppendOnlyMap: Spilling in-memory map of 484 MB to disk (1 time so far) What does this mean? How can I fix this? Due to this a small job takes forever. Mohit. P.S.: I am using