Re: High virtual memory consumption on spark-submit client.

2016-05-13 Thread jone
no, i have set master to yarn-cluster. when the sparkpi.running,the result of  free -t as follow [running]mqq@10.205.3.29:/data/home/hive/conf$ free -t total   used   free shared    buffers cached Mem:  32740732   32105684 635048  0 683332  

Re: High virtual memory consumption on spark-submit client.

2016-05-12 Thread Harsh J
How many CPU cores are on that machine? Read http://qr.ae/8Uv3Xq You can also confirm the above by running the pmap utility on your process and most of the virtual memory would be under 'anon'. On Fri, 13 May 2016 09:11 jone, wrote: > The virtual memory is 9G When i run

Re: High virtual memory consumption on spark-submit client.

2016-05-12 Thread Mich Talebzadeh
can you please do the following: jps|grep SparkSubmit| and send the output of ps aux|grep pid top -p PID and the output of free HTH Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

High virtual memory consumption on spark-submit client.

2016-05-12 Thread jone
The virtual memory is 9G When i run org.apache.spark.examples.SparkPi under yarn-cluster model,which using default configurations.   PID USER  PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND