Hi Vadim, Thank you so much this was a very useful command. This conversation is going on here
https://www.mail-archive.com/user@spark.apache.org/msg58656.html or you can just google " why spark driver program is creating so many threads? How can I limit this number? <https://www.mail-archive.com/search?l=user@spark.apache.org&q=subject:%22Re%5C%3A+why+spark+driver+program+is+creating+so+many+threads%5C%3F+How+can+I+limit+this+number%5C%3F%22&o=newest> " please take a look if you are interested. Thanks a lot! On Mon, Oct 31, 2016 at 8:14 AM, Vadim Semenov <vadim.seme...@datadoghq.com> wrote: > Have you tried to get number of threads in a running process using `cat > /proc/<pid>/status` ? > > On Sun, Oct 30, 2016 at 11:04 PM, kant kodali <kanth...@gmail.com> wrote: > >> yes I did run ps -ef | grep "app_name" and it is root. >> >> >> >> On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang <chin...@indetail.co.jp> >> wrote: >> >>> sorry, the UID >>> >>> On 10/31/16 11:59 AM, Chan Chor Pang wrote: >>> >>> actually if the max user processes is not the problem, i have no idea >>> >>> but i still suspecting the user, >>> as the user who run spark-submit is not necessary the pid for the JVM >>> process >>> >>> can u make sure when you "ps -ef | grep {your app id} " the PID is root? >>> On 10/31/16 11:21 AM, kant kodali wrote: >>> >>> The java process is run by the root and it has the same config >>> >>> sudo -i >>> >>> ulimit -a >>> >>> core file size (blocks, -c) 0 >>> data seg size (kbytes, -d) unlimited >>> scheduling priority (-e) 0 >>> file size (blocks, -f) unlimited >>> pending signals (-i) 120242 >>> max locked memory (kbytes, -l) 64 >>> max memory size (kbytes, -m) unlimited >>> open files (-n) 1024 >>> pipe size (512 bytes, -p) 8 >>> POSIX message queues (bytes, -q) 819200 >>> real-time priority (-r) 0 >>> stack size (kbytes, -s) 8192 >>> cpu time (seconds, -t) unlimited >>> max user processes (-u) 120242 >>> virtual memory (kbytes, -v) unlimited >>> file locks (-x) unlimited >>> >>> >>> >>> On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang <chin...@indetail.co.jp> >>> wrote: >>> >>>> I have the same Exception before and the problem fix after i change the >>>> nproc conf. >>>> >>>> > max user processes (-u) 120242 >>>> ↑this config does looks good. >>>> are u sure the user who run ulimit -a is the same user who run the Java >>>> process? >>>> depend on how u submit the job and your setting, spark job may execute >>>> by other user. >>>> >>>> >>>> On 10/31/16 10:38 AM, kant kodali wrote: >>>> >>>> when I did this >>>> >>>> cat /proc/sys/kernel/pid_max >>>> >>>> I got 32768 >>>> >>>> On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com> >>>> wrote: >>>> >>>>> I believe for ubuntu it is unlimited but I am not 100% sure (I just >>>>> read somewhere online). I ran ulimit -a and this is what I get >>>>> >>>>> core file size (blocks, -c) 0 >>>>> data seg size (kbytes, -d) unlimited >>>>> scheduling priority (-e) 0 >>>>> file size (blocks, -f) unlimited >>>>> pending signals (-i) 120242 >>>>> max locked memory (kbytes, -l) 64 >>>>> max memory size (kbytes, -m) unlimited >>>>> open files (-n) 1024 >>>>> pipe size (512 bytes, -p) 8 >>>>> POSIX message queues (bytes, -q) 819200 >>>>> real-time priority (-r) 0 >>>>> stack size (kbytes, -s) 8192 >>>>> cpu time (seconds, -t) unlimited >>>>> max user processes (-u) 120242 >>>>> virtual memory (kbytes, -v) unlimited >>>>> file locks (-x) unlimited >>>>> >>>>> On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang < >>>>> chin...@indetail.co.jp> wrote: >>>>> >>>>>> not sure for ubuntu, but i think you can just create the file by >>>>>> yourself >>>>>> the syntax will be the same as /etc/security/limits.conf >>>>>> >>>>>> nproc.conf not only limit java process but all process by the same >>>>>> user >>>>>> so even the jvm process does nothing, if the corresponding user is >>>>>> busy in other way >>>>>> the jvm process will still not able to create new thread. >>>>>> >>>>>> btw the default limit for centos is 1024 >>>>>> >>>>>> >>>>>> On 10/31/16 9:51 AM, kant kodali wrote: >>>>>> >>>>>> >>>>>> On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang < >>>>>> chin...@indetail.co.jp> wrote: >>>>>> >>>>>>> /etc/security/limits.d/90-nproc.conf >>>>>>> >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am using Ubuntu 16.04 LTS. I have this directory >>>>>> /etc/security/limits.d/ but I don't have any files underneath it. This >>>>>> error happens after running for 4 to 5 hours. I wonder if this is a GC >>>>>> issue? And I am thinking if I should use CMS. I have also posted this on >>>>>> SO >>>>>> since I havent got much response for this question >>>>>> http://stackoverflow.com/questions/40315589/dag-sch >>>>>> eduler-event-loop-java-lang-outofmemoryerror-unable-to-creat >>>>>> e-new-native >>>>>> >>>>>> >>>>>> Thanks, >>>>>> kant >>>>>> >>>>>> >>>>>> -- >>>>>> ---*------------------------------------------------*---*---*---*--- >>>>>> 株式会社INDETAIL >>>>>> ニアショア総合サービス事業本部 >>>>>> ゲームサービス事業部 >>>>>> 陳 楚鵬 >>>>>> E-mail :chin...@indetail.co.jp >>>>>> URL : http://www.indetail.co.jp >>>>>> >>>>>> 【札幌本社/LABO/LABO2】 >>>>>> 〒060-0042 >>>>>> 札幌市中央区大通西9丁目3番地33 >>>>>> キタコーセンタービルディング >>>>>> (札幌本社/LABO2:2階、LABO:9階) >>>>>> TEL:011-206-9235 FAX:011-206-9236 >>>>>> >>>>>> 【東京支店】 >>>>>> 〒108-0014 >>>>>> 東京都港区芝5丁目29番20号 クロスオフィス三田 >>>>>> TEL:03-6809-6502 FAX:03-6809-6504 >>>>>> >>>>>> 【名古屋サテライト】 >>>>>> 〒460-0002 >>>>>> 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD >>>>>> TEL:052-971-0086 >>>>>> >>>>>> >> >