The heap size of JVM can not been changed dynamically, so you
need to config it before running pyspark.
If you run it in local mode, you should config spark.driver.memory
(in 1.1 or master).
Or, you can use --driver-memory 2G (should work in 1.0+)
On Wed, Sep 10, 2014 at 10:43 PM, Mohit Singh
Hi,
I am using pyspark shell and am trying to create an rdd from numpy matrix
rdd = sc.parallelize(matrix)
I am getting the following error:
JVMDUMP039I Processing dump event systhrow, detail
java/lang/OutOfMemoryError at 2014/09/10 22:41:44 - please wait.
JVMDUMP032I JVM requested Heap dump
