yes I did run ps -ef | grep "app_name" and it is root.
On Sun, Oct 30, 2016 at 8:00 PM, Chan Chor Pang <chin...@indetail.co.jp> wrote: > sorry, the UID > > On 10/31/16 11:59 AM, Chan Chor Pang wrote: > > actually if the max user processes is not the problem, i have no idea > > but i still suspecting the user, > as the user who run spark-submit is not necessary the pid for the JVM > process > > can u make sure when you "ps -ef | grep {your app id} " the PID is root? > On 10/31/16 11:21 AM, kant kodali wrote: > > The java process is run by the root and it has the same config > > sudo -i > > ulimit -a > > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 120242 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 1024 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 120242 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > > > On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang <chin...@indetail.co.jp> > wrote: > >> I have the same Exception before and the problem fix after i change the >> nproc conf. >> >> > max user processes (-u) 120242 >> ↑this config does looks good. >> are u sure the user who run ulimit -a is the same user who run the Java >> process? >> depend on how u submit the job and your setting, spark job may execute by >> other user. >> >> >> On 10/31/16 10:38 AM, kant kodali wrote: >> >> when I did this >> >> cat /proc/sys/kernel/pid_max >> >> I got 32768 >> >> On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com> wrote: >> >>> I believe for ubuntu it is unlimited but I am not 100% sure (I just read >>> somewhere online). I ran ulimit -a and this is what I get >>> >>> core file size (blocks, -c) 0 >>> data seg size (kbytes, -d) unlimited >>> scheduling priority (-e) 0 >>> file size (blocks, -f) unlimited >>> pending signals (-i) 120242 >>> max locked memory (kbytes, -l) 64 >>> max memory size (kbytes, -m) unlimited >>> open files (-n) 1024 >>> pipe size (512 bytes, -p) 8 >>> POSIX message queues (bytes, -q) 819200 >>> real-time priority (-r) 0 >>> stack size (kbytes, -s) 8192 >>> cpu time (seconds, -t) unlimited >>> max user processes (-u) 120242 >>> virtual memory (kbytes, -v) unlimited >>> file locks (-x) unlimited >>> >>> On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang <chin...@indetail.co.jp> >>> wrote: >>> >>>> not sure for ubuntu, but i think you can just create the file by >>>> yourself >>>> the syntax will be the same as /etc/security/limits.conf >>>> >>>> nproc.conf not only limit java process but all process by the same user >>>> so even the jvm process does nothing, if the corresponding user is >>>> busy in other way >>>> the jvm process will still not able to create new thread. >>>> >>>> btw the default limit for centos is 1024 >>>> >>>> >>>> On 10/31/16 9:51 AM, kant kodali wrote: >>>> >>>> >>>> On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang <chin...@indetail.co.jp >>>> > wrote: >>>> >>>>> /etc/security/limits.d/90-nproc.conf >>>>> >>>> >>>> Hi, >>>> >>>> I am using Ubuntu 16.04 LTS. I have this directory >>>> /etc/security/limits.d/ but I don't have any files underneath it. This >>>> error happens after running for 4 to 5 hours. I wonder if this is a GC >>>> issue? And I am thinking if I should use CMS. I have also posted this on SO >>>> since I havent got much response for this question >>>> http://stackoverflow.com/questions/40315589/dag-sch >>>> eduler-event-loop-java-lang-outofmemoryerror-unable-to-creat >>>> e-new-native >>>> >>>> >>>> Thanks, >>>> kant >>>> >>>> >>>> -- >>>> ---*------------------------------------------------*---*---*---*--- >>>> 株式会社INDETAIL >>>> ニアショア総合サービス事業本部 >>>> ゲームサービス事業部 >>>> 陳 楚鵬 >>>> E-mail :chin...@indetail.co.jp >>>> URL : http://www.indetail.co.jp >>>> >>>> 【札幌本社/LABO/LABO2】 >>>> 〒060-0042 >>>> 札幌市中央区大通西9丁目3番地33 >>>> キタコーセンタービルディング >>>> (札幌本社/LABO2:2階、LABO:9階) >>>> TEL:011-206-9235 FAX:011-206-9236 >>>> >>>> 【東京支店】 >>>> 〒108-0014 >>>> 東京都港区芝5丁目29番20号 クロスオフィス三田 >>>> TEL:03-6809-6502 FAX:03-6809-6504 >>>> >>>> 【名古屋サテライト】 >>>> 〒460-0002 >>>> 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD >>>> TEL:052-971-0086 >>>> >>>>