Re: OutOfMemoryError when loading input file

2014-03-03 Thread Yonathan Perez
context: http://apache-spark-user-list.1001560.n3.nabble.com/OutOfMemoryError-when-loading-input-file-tp2213p2246.html Sent from the Apache Spark User List mailing list archive at Nabble.com.

OutOfMemoryError when loading input file

2014-03-01 Thread Yonathan Perez
Hello, I'm trying to run a simple test program that loads a large file (~12.4GB) into memory of a single many-core machine. The machine I'm using has more than enough memory (1TB RAM) and 64 cores (of which I want to use 16 for worker threads). Even though I set both the executor memory (spark.exe