unsubscribe

Re: What influences the space complexity of Spark operations?

Submitted: SPARK-14389 - OOM during BroadcastNestedLoopJoin. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/What-influences-the-space-complexity-of-Spark-operations-tp16944p17029.html Sent from the Apache Spark Developers List mailing list archive at

What influences the space complexity of Spark operations?

*What we’ve observed* Increasing the number of partitions (and thus decreasing the partition size) seems to reliably help avoid OOM errors. To demonstrate this we used a single executor and loaded a small table into a DataFrame, persisted it with MEMORY_AND_DISK, repartitioned it and joined it to

Re: OOM and "spark.buffer.pageSize"

Yes I have. That’s the best source of information at the moment. Thanks. -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/OOM-and-spark-buffer-pageSize-tp16890p16892.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

OOM and "spark.buffer.pageSize"

I'm attempting to address an OOM issue. I saw referenced in java.lang.OutOfMemoryError: Unable to acquire bytes of memory the configuration setting