Have you tried to get number of threads in a running process using `cat /proc/<pid>/status` ?
On Sun, Oct 30, 2016 at 11:04 PM, kant kodali <kanth...@gmail.com> wrote: > yes I did run ps -ef | grep "app_name" and it is root. > > > > On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang <chin...@indetail.co.jp> > wrote: > >> sorry, the UID >> >> On 10/31/16 11:59 AM, Chan Chor Pang wrote: >> >> actually if the max user processes is not the problem, i have no idea >> >> but i still suspecting the user, >> as the user who run spark-submit is not necessary the pid for the JVM >> process >> >> can u make sure when you "ps -ef | grep {your app id} " the PID is root? >> On 10/31/16 11:21 AM, kant kodali wrote: >> >> The java process is run by the root and it has the same config >> >> sudo -i >> >> ulimit -a >> >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 120242 >> max locked memory (kbytes, -l) 64 >> max memory size (kbytes, -m) unlimited >> open files (-n) 1024 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 8192 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 120242 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> >> >> >> On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang <chin...@indetail.co.jp> >> wrote: >> >>> I have the same Exception before and the problem fix after i change the >>> nproc conf. >>> >>> > max user processes (-u) 120242 >>> ↑this config does looks good. >>> are u sure the user who run ulimit -a is the same user who run the Java >>> process? >>> depend on how u submit the job and your setting, spark job may execute >>> by other user. >>> >>> >>> On 10/31/16 10:38 AM, kant kodali wrote: >>> >>> when I did this >>> >>> cat /proc/sys/kernel/pid_max >>> >>> I got 32768 >>> >>> On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com> wrote: >>> >>>> I believe for ubuntu it is unlimited but I am not 100% sure (I just >>>> read somewhere online). I ran ulimit -a and this is what I get >>>> >>>> core file size (blocks, -c) 0 >>>> data seg size (kbytes, -d) unlimited >>>> scheduling priority (-e) 0 >>>> file size (blocks, -f) unlimited >>>> pending signals (-i) 120242 >>>> max locked memory (kbytes, -l) 64 >>>> max memory size (kbytes, -m) unlimited >>>> open files (-n) 1024 >>>> pipe size (512 bytes, -p) 8 >>>> POSIX message queues (bytes, -q) 819200 >>>> real-time priority (-r) 0 >>>> stack size (kbytes, -s) 8192 >>>> cpu time (seconds, -t) unlimited >>>> max user processes (-u) 120242 >>>> virtual memory (kbytes, -v) unlimited >>>> file locks (-x) unlimited >>>> >>>> On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang <chin...@indetail.co.jp >>>> > wrote: >>>> >>>>> not sure for ubuntu, but i think you can just create the file by >>>>> yourself >>>>> the syntax will be the same as /etc/security/limits.conf >>>>> >>>>> nproc.conf not only limit java process but all process by the same user >>>>> so even the jvm process does nothing, if the corresponding user is >>>>> busy in other way >>>>> the jvm process will still not able to create new thread. >>>>> >>>>> btw the default limit for centos is 1024 >>>>> >>>>> >>>>> On 10/31/16 9:51 AM, kant kodali wrote: >>>>> >>>>> >>>>> On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang < >>>>> chin...@indetail.co.jp> wrote: >>>>> >>>>>> /etc/security/limits.d/90-nproc.conf >>>>>> >>>>> >>>>> Hi, >>>>> >>>>> I am using Ubuntu 16.04 LTS. I have this directory >>>>> /etc/security/limits.d/ but I don't have any files underneath it. This >>>>> error happens after running for 4 to 5 hours. I wonder if this is a GC >>>>> issue? And I am thinking if I should use CMS. I have also posted this on >>>>> SO >>>>> since I havent got much response for this question >>>>> http://stackoverflow.com/questions/40315589/dag-sch >>>>> eduler-event-loop-java-lang-outofmemoryerror-unable-to-creat >>>>> e-new-native >>>>> >>>>> >>>>> Thanks, >>>>> kant >>>>> >>>>> >>>>> -- >>>>> ---*------------------------------------------------*---*---*---*--- >>>>> 株式会社INDETAIL >>>>> ニアショア総合サービス事業本部 >>>>> ゲームサービス事業部 >>>>> 陳 楚鵬 >>>>> E-mail :chin...@indetail.co.jp >>>>> URL : http://www.indetail.co.jp >>>>> >>>>> 【札幌本社/LABO/LABO2】 >>>>> 〒060-0042 >>>>> 札幌市中央区大通西9丁目3番地33 >>>>> キタコーセンタービルディング >>>>> (札幌本社/LABO2:2階、LABO:9階) >>>>> TEL:011-206-9235 FAX:011-206-9236 >>>>> >>>>> 【東京支店】 >>>>> 〒108-0014 >>>>> 東京都港区芝5丁目29番20号 クロスオフィス三田 >>>>> TEL:03-6809-6502 FAX:03-6809-6504 >>>>> >>>>> 【名古屋サテライト】 >>>>> 〒460-0002 >>>>> 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD >>>>> TEL:052-971-0086 >>>>> >>>>> >