Hi Mich,
I am able to write the files to storage after adding extra parameter.
FYI..
This one I used.
spark.sql.autoBroadcastJoinThreshold="-1"
On Mon, Oct 24, 2016 at 7:22 PM, Mich Talebzadeh <mich.talebza...@gmail.com>
wrote:
> Rather strange as you have plenty free memory there.
>
> Try reducing driver memory to 2GB and executer memory to 2GB and run it
> again
>
> ${SPARK_HOME}/bin/spark-submit \
> --driver-memory 2G \
> --num-executors 2 \
> --executor-cores 1 \
> --executor-memory 2G \
> --master spark://IPAddress:7077 \
>
> HTH
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 24 October 2016 at 13:15, Sankar Mittapally <sankar.mittapally@
> creditvidya.com> wrote:
>
>> Hi Mich,
>>
>> Yes, I am using standalone mode cluster, We have two executors with 10G
>> memory each. We have two workers.
>>
>> FYI..
>>
>>
>>
>> On Mon, Oct 24, 2016 at 5:22 PM, Mich Talebzadeh <
>> mich.talebza...@gmail.com> wrote:
>>
>>> Sounds like you are running in standalone mode.
>>>
>>> Have you checked the UI on port 4040 (default) to see where memory is
>>> going. Why do you need executor memory of 10GB?
>>>
>>> How many executors are running and plus how many slaves started?
>>>
>>> In standalone mode executors run on workers (UI 8080)
>>>
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>>
>>> LinkedIn *
>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 24 October 2016 at 12:19, sankarmittapally <
>>> sankar.mittapa...@creditvidya.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a three node cluster with 30G of Memory. I am trying to
>>>> analyzing
>>>> the data of 200MB and running out of memory every time. This is the
>>>> command
>>>> I am using
>>>>
>>>> Driver Memory = 10G
>>>> Executor memory=10G
>>>>
>>>> sc <- sparkR.session(master =
>>>> "spark://ip-172-31-6-116:7077",sparkConfig=list(spark.execut
>>>> or.memory="10g",spark.app.name="Testing",spark.driver.memory
>>>> ="14g",spark.executor.extraJavaOption="-Xms2g
>>>> -Xmx5g -XX:MaxPermSize=1024M",spark.driver.extraJavaOption="-Xms2g
>>>> -Xmx5g
>>>> -XX:MaxPermSize=1024M",spark.cores.max="2"))
>>>>
>>>>
>>>> [D 16:43:51.437 NotebookApp] 200 GET
>>>> /api/contents?type=directory&_=1477289197671 (123.176.38.226) 7.96ms
>>>> Exception in thread "broadcast-exchange-0" java.lang.OutOfMemoryError:
>>>> Java
>>>> heap space
>>>> at
>>>> org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.appe
>>>> nd(HashedRelation.scala:539)
>>>> at
>>>> org.apache.spark.sql.execution.joins.LongHashedRelation$.app
>>>> ly(HashedRelation.scala:803)
>>>> at
>>>> org.apache.spark.sql.execution.joins.HashedRelation$.apply(H
>>>> ashedRelation.scala:105)
>>>> at
>>>> org.apache.spark.sql.execution.joins.HashedRelationBroadcast
>>>> Mode.transform(HashedRelation.scala:816)
>>>> at
>>>> org.apache.spark.sql.execution.joins.HashedRelationBroadcast
>>>> Mode.transform(HashedRelation.scala:812)
>>>> at
>>>> org.apache.spark.sql.execution.exchange.BroadcastExchangeExe
>>>> c$$anonfun$relationFuture$1$$anonfun$apply$1.apply(Broadcast
>>>> ExchangeExec.
>>>> scala:90)
>>>> at
>>>> org.apache.spark.sql.execution.exchange.BroadcastExchangeExe
>>>> c$$anonfun$relationFuture$1$$anonfun$apply$1.apply(Broadcast
>>>> ExchangeExec.
>>>> scala:72)
>>>> at
>>>> org.apache.spark.sql.execution.SQLExecution$.withExecutionId
>>>> (SQLExecution.scala:94)
>>>> at
>>>> org.apache.spark.sql.execution.exchange.BroadcastExchangeExe
>>>> c$$anonfun$relationFuture$1.apply(BroadcastExchangeExec.scala:72)
>>>> at
>>>> org.apache.spark.sql.execution.exchange.BroadcastExchangeExe
>>>> c$$anonfun$relationFuture$1.apply(BroadcastExchangeExec.scala:72)
>>>> at
>>>> scala.concurrent.impl.Future$PromiseCompletingRunnable.lifte
>>>> dTree1$1(Future.scala:24)
>>>> at
>>>> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(F
>>>> uture.scala:24)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>> Executor.java:1142)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>> lExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-spark-user-list.
>>>> 1001560.n3.nabble.com/JAVA-heap-space-issue-tp27950.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>>>>
>>>>
>>>
>>
>
