Can you show me at Spark UI -> executors tab and storage tab. It will show us how many executor was executed and how much memory we use to cache.
> On Jul 14, 2016, at 9:49 AM, Jean Georges Perrin <j...@jgp.net> wrote: > > I use it as a standalone cluster. > > I run it through start-master, then start-slave. I only have one slave now, > but I will probably have a few soon. > > The "application" is run on a separate box. > > When everything was running on my mac, i was in local mode, but i never setup > anything in local mode. Going "production" was a little more complex that I > thought. > >> On Jul 13, 2016, at 10:35 PM, Chanh Le <giaosu...@gmail.com >> <mailto:giaosu...@gmail.com>> wrote: >> >> Hi Jean, >> How do you run your Spark Application? Local Mode, Cluster Mode? >> If you run in local mode did you use —driver-memory and —executor-memory >> because in local mode your setting about executor and driver didn’t work >> that you expected. >> >> >> >> >>> On Jul 14, 2016, at 8:43 AM, Jean Georges Perrin <j...@jgp.net >>> <mailto:j...@jgp.net>> wrote: >>> >>> Looks like replacing the setExecutorEnv() by set() did the trick... let's >>> see how fast it'll process my 50x 10ˆ15 data points... >>> >>>> On Jul 13, 2016, at 9:24 PM, Jean Georges Perrin <j...@jgp.net >>>> <mailto:j...@jgp.net>> wrote: >>>> >>>> I have added: >>>> >>>> SparkConf conf = new >>>> SparkConf().setAppName("app").setExecutorEnv("spark.executor.memory", "8g") >>>> .setMaster("spark://10.0.100.120:7077 >>>> <spark://10.0.100.120:7077>"); >>>> >>>> but it did not change a thing >>>> >>>>> On Jul 13, 2016, at 9:14 PM, Jean Georges Perrin <j...@jgp.net >>>>> <mailto:j...@jgp.net>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have a Java memory issue with Spark. The same application working on my >>>>> 8GB Mac crashes on my 72GB Ubuntu server... >>>>> >>>>> I have changed things in the conf file, but it looks like Spark does not >>>>> care, so I wonder if my issues are with the driver or executor. >>>>> >>>>> I set: >>>>> >>>>> spark.driver.memory 20g >>>>> spark.executor.memory 20g >>>>> And, whatever I do, the crash is always at the same spot in the app, >>>>> which makes me think that it is a driver problem. >>>>> >>>>> The exception I get is: >>>>> >>>>> 16/07/13 20:36:30 WARN TaskSetManager: Lost task 0.0 in stage 7.0 (TID >>>>> 208, micha.nc.rr.com): java.lang.OutOfMemoryError: Java heap space >>>>> at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57) >>>>> at java.nio.CharBuffer.allocate(CharBuffer.java:335) >>>>> at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:810) >>>>> at org.apache.hadoop.io.Text.decode(Text.java:412) >>>>> at org.apache.hadoop.io.Text.decode(Text.java:389) >>>>> at org.apache.hadoop.io.Text.toString(Text.java:280) >>>>> at >>>>> org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105) >>>>> at >>>>> org.apache.spark.sql.execution.datasources.json.JSONRelation$$anonfun$org$apache$spark$sql$execution$datasources$json$JSONRelation$$createBaseRdd$1.apply(JSONRelation.scala:105) >>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>>> at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) >>>>> at scala.collection.Iterator$class.foreach(Iterator.scala:727) >>>>> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) >>>>> at >>>>> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:144) >>>>> at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1157) >>>>> at >>>>> scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:201) >>>>> at scala.collection.AbstractIterator.aggregate(Iterator.scala:1157) >>>>> at >>>>> org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135) >>>>> at >>>>> org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1135) >>>>> at >>>>> org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136) >>>>> at >>>>> org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136) >>>>> at >>>>> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) >>>>> at >>>>> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710) >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:270) >>>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:89) >>>>> at >>>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> >>>>> I have set a small memory "dumper" in my app. At the beginning, it says: >>>>> >>>>> ** Free ......... 1,413,566 >>>>> ** Allocated .... 1,705,984 >>>>> ** Max .......... 16,495,104 >>>>> **> Total free ... 16,202,686 >>>>> Just before the crash, it says: >>>>> >>>>> ** Free ......... 1,461,633 >>>>> ** Allocated .... 1,786,880 >>>>> ** Max .......... 16,495,104 >>>>> **> Total free ... 16,169,857 >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
