Pete - can you do me a favor? https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala#L174
Print the parameters that are passed into the getPageSize function, and check their values. On Mon, Sep 14, 2015 at 4:32 PM, Reynold Xin <r...@databricks.com> wrote: > Is this on latest master / branch-1.5? > > out of the box we reserve only 16% (0.2 * 0.8) of the memory for execution > (e.g. aggregate, join) / shuffle sorting. With a 3GB heap, that's 480MB. So > each task gets 480MB / 32 = 15MB, and each operator reserves at least one > page for execution. If your page size is 4MB, it only takes 3 operators to > use up its memory. > > The thing is page size is dynamically determined -- and in your case it > should be smaller than 4MB. > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/shuffle/ShuffleMemoryManager.scala#L174 > > Maybe there is a place that in the maven tests that we explicitly set the > page size (spark.buffer.pageSize) to 4MB? If yes, we need to find it and > just remove it. > > > On Mon, Sep 14, 2015 at 4:16 AM, Pete Robbins <robbin...@gmail.com> wrote: > >> I keep hitting errors running the tests on 1.5 such as >> >> >> - join31 *** FAILED *** >> Failed to execute query using catalyst: >> Error: Job aborted due to stage failure: Task 9 in stage 3653.0 failed >> 1 times, most recent failure: Lost task 9.0 in stage 3653.0 (TID 123363, >> localhost): java.io.IOException: Unable to acquire 4194304 bytes of memory >> at >> org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPage(UnsafeExternalSorter.java:368) >> >> >> This is using the command >> build/mvn -Pyarn -Phadoop-2.2 -Phive -Phive-thriftserver test >> >> >> I don't see these errors in any of the amplab jenkins builds. Do those >> builds have any configuration/environment that I may be missing? My build >> is running with whatever defaults are in the top level pom.xml, eg -Xmx3G. >> >> I can make these tests pass by setting spark.shuffle.memoryFraction=0.6 >> in the HiveCompatibilitySuite rather than the default 0.2 value. >> >> Trying to analyze what is going on with the test it is related to the >> number of active tasks, which seems to rise to 32, and so the >> ShuffleMemoryManager allows less memory per task even though most of those >> tasks do not have any memory allocated to them. >> >> Has anyone seen issues like this before? >> > >
