Just want to see if this helps. Are you doing heavy collects and persist that? If that is so, you might want to parallelize that collection by converting to an RDD.
Thanks, RK On Tue, Jul 19, 2016 at 12:09 AM, Saurav Sinha <sauravsinh...@gmail.com> wrote: > Hi Mich, > > 1. In what mode are you running the spark standalone, yarn-client, > yarn cluster etc > > Ans: spark standalone > > 1. You have 4 nodes with each executor having 10G. How many actual > executors do you see in UI (Port 4040 by default) > > Ans: There are 4 executor on which am using 8 cores (--total-executor-core > 32) > > 1. What is master memory? Are you referring to diver memory? May be I > am misunderstanding this > > Ans: Driver memory is set as --drive-memory 5g > > 1. The only real correlation I see with the driver memory is when you > are running in local mode where worker lives within JVM process that you > start with spark-shell etc. In that case driver memory matters. However, it > appears that you are running in another mode with 4 nodes? > > Ans: I am running my job as spark-submit and on my worker(executor) node > there is no OOM issue ,it only happening on driver app. > > Thanks, > Saurav Sinha > > On Tue, Jul 19, 2016 at 2:42 AM, Mich Talebzadeh < > mich.talebza...@gmail.com> wrote: > >> can you please clarify: >> >> >> 1. In what mode are you running the spark standalone, yarn-client, >> yarn cluster etc >> 2. You have 4 nodes with each executor having 10G. How many actual >> executors do you see in UI (Port 4040 by default) >> 3. What is master memory? Are you referring to diver memory? May be I >> am misunderstanding this >> 4. The only real correlation I see with the driver memory is when you >> are running in local mode where worker lives within JVM process that you >> start with spark-shell etc. In that case driver memory matters. However, >> it >> appears that you are running in another mode with 4 nodes? >> >> Can you get a snapshot of your environment tab in UI and send the output >> please? >> >> HTH >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any loss, damage or destruction of data or any other property which may >> arise from relying on this email's technical content is explicitly >> disclaimed. The author will in no case be liable for any monetary damages >> arising from such loss, damage or destruction. >> >> >> >> On 18 July 2016 at 11:50, Saurav Sinha <sauravsinh...@gmail.com> wrote: >> >>> I have set --drive-memory 5g. I need to understand that as no of >>> partition increase drive-memory need to be increased. What will be best >>> ration of No of partition/drive-memory. >>> >>> On Mon, Jul 18, 2016 at 4:07 PM, Zhiliang Zhu <zchl.j...@yahoo.com> >>> wrote: >>> >>>> try to set --drive-memory xg , x would be as large as can be set . >>>> >>>> >>>> On Monday, July 18, 2016 6:31 PM, Saurav Sinha <sauravsinh...@gmail.com> >>>> wrote: >>>> >>>> >>>> Hi, >>>> >>>> I am running spark job. >>>> >>>> Master memory - 5G >>>> executor memort 10G(running on 4 node) >>>> >>>> My job is getting killed as no of partition increase to 20K. >>>> >>>> 16/07/18 14:53:13 INFO DAGScheduler: Got job 17 (foreachPartition at >>>> WriteToKafka.java:45) with 13524 output partitions (allowLocal=false) >>>> 16/07/18 14:53:13 INFO DAGScheduler: Final stage: ResultStage >>>> 640(foreachPartition at WriteToKafka.java:45) >>>> 16/07/18 14:53:13 INFO DAGScheduler: Parents of final stage: >>>> List(ShuffleMapStage 518, ShuffleMapStage 639) >>>> 16/07/18 14:53:23 INFO DAGScheduler: Missing parents: List() >>>> 16/07/18 14:53:23 INFO DAGScheduler: Submitting ResultStage 640 >>>> (MapPartitionsRDD[271] at map at BuildSolrDocs.java:209), which has no >>>> missing >>>> parents >>>> 16/07/18 14:53:23 INFO MemoryStore: ensureFreeSpace(8248) called with >>>> curMem=41923262, maxMem=2778778828 >>>> 16/07/18 14:53:23 INFO MemoryStore: Block broadcast_90 stored as values >>>> in memory (estimated size 8.1 KB, free 2.5 GB) >>>> Exception in thread "dag-scheduler-event-loop" >>>> java.lang.OutOfMemoryError: Java heap space >>>> at >>>> org.apache.spark.util.io.ByteArrayChunkOutputStream.allocateNewChunkIfNeeded(ByteArrayChunkOutputStream.scala:66) >>>> at >>>> org.apache.spark.util.io.ByteArrayChunkOutputStream.write(ByteArrayChunkOutputStream.scala:55) >>>> at >>>> org.xerial.snappy.SnappyOutputStream.dumpOutput(SnappyOutputStream.java:294) >>>> at >>>> org.xerial.snappy.SnappyOutputStream.flush(SnappyOutputStream.java:273) >>>> at >>>> org.apache.spark.io.SnappyOutputStreamWrapper.flush(CompressionCodec.scala:197) >>>> at >>>> java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1822) >>>> >>>> >>>> Help needed. >>>> >>>> -- >>>> Thanks and Regards, >>>> >>>> Saurav Sinha >>>> >>>> Contact: 9742879062 >>>> >>>> >>>> >>> >>> >>> -- >>> Thanks and Regards, >>> >>> Saurav Sinha >>> >>> Contact: 9742879062 >>> >> >> > > > -- > Thanks and Regards, > > Saurav Sinha > > Contact: 9742879062 > -- Collective[i] dramatically improves sales and marketing performance using technology, applications and a revolutionary network designed to provide next generation analytics and decision-support directly to business users. Our goal is to maximize human potential and minimize mistakes. In most cases, the results are astounding. We cannot, however, stop emails from sometimes being sent to the wrong person. If you are not the intended recipient, please notify us by replying to this email's sender and deleting it (and any attachments) permanently from your system. If you are, please respect the confidentiality of this communication's contents.
