Hi Nezih, I just reported a somewhat similar issue, and I have a potential fix -- SPARK-14560, looks like you are already watching it :). You can try out that patch, you have to explicitly enable the change in behavior with "spark.shuffle.spillAfterRead=true". Honestly, I don't think these issues are the same, as I've always seen that case lead to acquiring 0 bytes, while in your case you are requesting GBs and getting something pretty close, so my hunch is that it is different ... but might be worth a shot to see if it is the issue.
Turning on debug logging for TaskMemoryManager might help track the root cause -- you'll get information on which consumers are using memory and when there are spill attempts. (Note that even if the patch I have for SPARK-14560 doesn't fix your issue, it might still make those debug logs a bit more clear, since it'll report memory used by Spillables.) Imran On Mon, Apr 4, 2016 at 10:52 PM, Nezih Yigitbasi < nyigitb...@netflix.com.invalid> wrote: > Nope, I didn't have a chance to track the root cause, and IIRC we didn't > observe it when dyn. alloc. is off. > > On Mon, Apr 4, 2016 at 6:16 PM Reynold Xin <r...@databricks.com> wrote: > >> BTW do you still see this when dynamic allocation is off? >> >> On Mon, Apr 4, 2016 at 6:16 PM, Reynold Xin <r...@databricks.com> wrote: >> >>> Nezih, >>> >>> Have you had a chance to figure out why this is happening? >>> >>> >>> On Tue, Mar 22, 2016 at 1:32 AM, james <yiaz...@gmail.com> wrote: >>> >>>> I guess different workload cause diff result ? >>>> >>>> >>>> >>>> -- >>>> View this message in context: >>>> http://apache-spark-developers-list.1001551.n3.nabble.com/java-lang-OutOfMemoryError-Unable-to-acquire-bytes-of-memory-tp16773p16789.html >>>> Sent from the Apache Spark Developers List mailing list archive at >>>> Nabble.com. >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>> >>>> >>> >>
