Long pauses after writing to sequence files

We have a series of spark jobs which run in succession over various cached datasets, do small groups and transforms, and then call saveAsSequenceFile() on them. Each call to save as a sequence file appears to have done its work, the task says it completed in "xxx.x seconds" but then it pauses

Re: Long pauses after writing to sequence files

Not quiet sure, but it could be the GC Pause, if you are holding too much objects in memory. You can check this tuning part if you haven't already been through it. Thanks Best Regards On Sat, Jan 31, 2015 at 7:22 AM, Corey Nolet wrote: > We have