Hi Michael Campbell, Are you deploying against yarn or standalone mode? In yarn try setting the shell variables SPARK_EXECUTOR_MEMORY=2G in standalone try and set SPARK_WORKER_MEMORY=2G.
Cheers, Holden :) On Thu, Oct 16, 2014 at 2:22 PM, Michael Campbell < michael.campb...@gmail.com> wrote: > TL;DR - a spark SQL job fails with an OOM (Out of heap space) error. If > given "--executor-memory" values, it won't even start. Even (!) if the > values given ARE THE SAME AS THE DEFAULT. > > > > Without --executor-memory: > > 14/10/16 17:14:58 INFO TaskSetManager: Serialized task 1.0:64 as 14710 > bytes in 1 ms > 14/10/16 17:14:58 WARN TaskSetManager: Lost TID 26 (task 1.0:25) > 14/10/16 17:14:58 WARN TaskSetManager: Loss was due to > java.lang.OutOfMemoryError > java.lang.OutOfMemoryError: Java heap space > at > parquet.hadoop.ParquetFileReader$ConsecutiveChunkList.readAll(ParquetFileReader.java:609) > at > parquet.hadoop.ParquetFileReader.readNextRowGroup(ParquetFileReader.java:360) > ... > > > USING --executor-memory (WITH ANY VALUE), even "1G" which is the default: > > Parsed arguments: > master spark://<redacted>:7077 > deployMode null > executorMemory 1G > ... > > System properties: > spark.executor.memory -> 1G > spark.eventLog.enabled -> true > ... > > 14/10/16 17:14:23 INFO TaskSchedulerImpl: Adding task set 1.0 with 678 > tasks > 14/10/16 17:14:38 WARN TaskSchedulerImpl: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient memory > > > > Spark 1.0.0. Is this a bug? > > -- Cell : 425-233-8271
