Re: java.lang.OutOfMemoryError: unable to create new native thread
sorry, the UID
On 10/31/16 11:59 AM, Chan Chor Pang wrote:
actually if the max user processes is not the problem, i have no idea
but i still suspecting the user,
as the user who run spark-submit is not necessary the pid for the JVM
process
can u make sure when you "ps -ef | grep {your app id} " the PID is root?
On 10/31/16 11:21 AM, kant kodali wrote:
The java process is run by the root and it has the same config
sudo -i
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 120242
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 120242
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang
<chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote:
I have the same Exception before and the problem fix after i
change the nproc conf.
> max user processes (-u) 120242
↑this config does looks good.
are u sure the user who run ulimit -a is the same user who run
the Java process?
depend on how u submit the job and your setting, spark job may
execute by other user.
On 10/31/16 10:38 AM, kant kodali wrote:
when I did this
cat /proc/sys/kernel/pid_max
I got 32768
On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com
<mailto:kanth...@gmail.com>> wrote:
I believe for ubuntu it is unlimited but I am not 100% sure
(I just read somewhere online). I ran ulimit -a and this is
what I get
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 120242
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 120242
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang
<chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote:
not sure for ubuntu, but i think you can just create the
file by yourself
the syntax will be the same as /etc/security/limits.conf
nproc.conf not only limit java process but all process
by the same user
so even the jvm process does nothing, if the
corresponding user is busy in other way
the jvm process will still not able to create new thread.
btw the default limit for centos is 1024
On 10/31/16 9:51 AM, kant kodali wrote:
On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang
<chin...@indetail.co.jp
<mailto:chin...@indetail.co.jp>> wrote:
/etc/security/limits.d/90-nproc.conf
Hi,
I am using Ubuntu 16.04 LTS. I have this directory
/etc/security/limits.d/ but I don't have any files
underneath it. This error happens after running for 4
to 5 hours. I wonder if this is a GC issue? And I am
thinking if I should use CMS. I have also posted this
on SO since I havent got much response for this
question
http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native
<http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native>
Thanks,
kant
--
---**---*---*---*---
株式会社INDETAIL
ニアショア総合サービス事業本部
ゲームサービス事業部
陳 楚鵬
E-mail :chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>
URL :http://www.indetail.co.jp
【札幌本社／LABO／LABO2】
〒060-0042
札幌市中央区大通西9丁目3番地33
キタコーセンタービルディング
（札幌本社／LABO2：2階、LABO：9階）
TEL：011-206-9235 FAX：011-206-9236
【東京支店】
〒108-0014
東京都港区芝5丁目29番20号
Re: java.lang.OutOfMemoryError: unable to create new native thread
actually if the max user processes is not the problem, i have no idea
but i still suspecting the user,
as the user who run spark-submit is not necessary the pid for the JVM
process
can u make sure when you "ps -ef | grep {your app id} " the PID is root?
On 10/31/16 11:21 AM, kant kodali wrote:
The java process is run by the root and it has the same config
sudo -i
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 120242
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 120242
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
On Sun, Oct 30, 2016 at 7:01 PM, Chan Chor Pang
<chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote:
I have the same Exception before and the problem fix after i
change the nproc conf.
> max user processes (-u) 120242
↑this config does looks good.
are u sure the user who run ulimit -a is the same user who run the
Java process?
depend on how u submit the job and your setting, spark job may
execute by other user.
On 10/31/16 10:38 AM, kant kodali wrote:
when I did this
cat /proc/sys/kernel/pid_max
I got 32768
On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com
<mailto:kanth...@gmail.com>> wrote:
I believe for ubuntu it is unlimited but I am not 100% sure
(I just read somewhere online). I ran ulimit -a and this is
what I get
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 120242
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 120242
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang
<chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote:
not sure for ubuntu, but i think you can just create the
file by yourself
the syntax will be the same as /etc/security/limits.conf
nproc.conf not only limit java process but all process by
the same user
so even the jvm process does nothing, if the
corresponding user is busy in other way
the jvm process will still not able to create new thread.
btw the default limit for centos is 1024
On 10/31/16 9:51 AM, kant kodali wrote:
On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang
<chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>>
wrote:
/etc/security/limits.d/90-nproc.conf
Hi,
I am using Ubuntu 16.04 LTS. I have this directory
/etc/security/limits.d/ but I don't have any files
underneath it. This error happens after running for 4 to
5 hours. I wonder if this is a GC issue? And I am
thinking if I should use CMS. I have also posted this on
SO since I havent got much response for this question
http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native
<http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native>
Thanks,
kant
--
---**---*---*---*---
株式会社INDETAIL
ニアショア総合サービス事業本部
ゲームサービス事業部
陳 楚鵬
E-mail :chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>
URL :http://www.indetail.co.jp
【札幌本社／LABO／LABO2】
〒060-0042
札幌市中央区大通西9丁目3番地33
キタコーセンタービルディング
（札幌本社／LABO2：2階、LABO：9階）
TEL：011-206-9235 FAX：011-206-9236
【東京支店】
〒108-0014
Re: java.lang.OutOfMemoryError: unable to create new native thread
I have the same Exception before and the problem fix after i change the nproc conf. > max user processes (-u) 120242 ↑this config does looks good. are u sure the user who run ulimit -a is the same user who run the Java process? depend on how u submit the job and your setting, spark job may execute by other user. On 10/31/16 10:38 AM, kant kodali wrote: when I did this cat /proc/sys/kernel/pid_max I got 32768 On Sun, Oct 30, 2016 at 6:36 PM, kant kodali <kanth...@gmail.com <mailto:kanth...@gmail.com>> wrote: I believe for ubuntu it is unlimited but I am not 100% sure (I just read somewhere online). I ran ulimit -a and this is what I get core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 120242 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 120242 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited On Sun, Oct 30, 2016 at 6:15 PM, Chan Chor Pang <chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote: not sure for ubuntu, but i think you can just create the file by yourself the syntax will be the same as /etc/security/limits.conf nproc.conf not only limit java process but all process by the same user so even the jvm process does nothing, if the corresponding user is busy in other way the jvm process will still not able to create new thread. btw the default limit for centos is 1024 On 10/31/16 9:51 AM, kant kodali wrote: On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang <chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote: /etc/security/limits.d/90-nproc.conf Hi, I am using Ubuntu 16.04 LTS. I have this directory /etc/security/limits.d/ but I don't have any files underneath it. This error happens after running for 4 to 5 hours. I wonder if this is a GC issue? And I am thinking if I should use CMS. I have also posted this on SO since I havent got much response for this question http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native <http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native> Thanks, kant -- ---**---*---*---*--- 株式会社INDETAIL ニアショア総合サービス事業本部 ゲームサービス事業部 陳 楚鵬 E-mail :chin...@indetail.co.jp <mailto:chin...@indetail.co.jp> URL :http://www.indetail.co.jp 【札幌本社／LABO／LABO2】 〒060-0042 札幌市中央区大通西9丁目3番地33 キタコーセンタービルディング （札幌本社／LABO2：2階、LABO：9階） TEL：011-206-9235 FAX：011-206-9236 【東京支店】 〒108-0014 東京都港区芝5丁目29番20号 クロスオフィス三田 TEL：03-6809-6502 FAX：03-6809-6504 【名古屋サテライト】 〒460-0002 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD TEL：052-971-0086
Re: java.lang.OutOfMemoryError: unable to create new native thread
not sure for ubuntu, but i think you can just create the file by yourself the syntax will be the same as /etc/security/limits.conf nproc.conf not only limit java process but all process by the same user so even the jvm process does nothing, if the corresponding user is busy in other way the jvm process will still not able to create new thread. btw the default limit for centos is 1024 On 10/31/16 9:51 AM, kant kodali wrote: On Sun, Oct 30, 2016 at 5:22 PM, Chan Chor Pang <chin...@indetail.co.jp <mailto:chin...@indetail.co.jp>> wrote: /etc/security/limits.d/90-nproc.conf Hi, I am using Ubuntu 16.04 LTS. I have this directory /etc/security/limits.d/ but I don't have any files underneath it. This error happens after running for 4 to 5 hours. I wonder if this is a GC issue? And I am thinking if I should use CMS. I have also posted this on SO since I havent got much response for this question http://stackoverflow.com/questions/40315589/dag-scheduler-event-loop-java-lang-outofmemoryerror-unable-to-create-new-native Thanks, kant -- ---**---*---*---*--- 株式会社INDETAIL ニアショア総合サービス事業本部 ゲームサービス事業部 陳 楚鵬 E-mail :chin...@indetail.co.jp URL : http://www.indetail.co.jp 【札幌本社／LABO／LABO2】 〒060-0042 札幌市中央区大通西9丁目3番地33 キタコーセンタービルディング （札幌本社／LABO2：2階、LABO：9階） TEL：011-206-9235 FAX：011-206-9236 【東京支店】 〒108-0014 東京都港区芝5丁目29番20号 クロスオフィス三田 TEL：03-6809-6502 FAX：03-6809-6504 【名古屋サテライト】 〒460-0002 愛知県名古屋市中区丸の内3丁目17番24号 NAYUTA BLD TEL：052-971-0086
Re: java.lang.OutOfMemoryError: unable to create new native thread
you may want to check the process limit of the user who responsible for starting the JVM. /etc/security/limits.d/90-nproc.conf On 10/29/16 4:47 AM, kant kodali wrote: "dag-scheduler-event-loop" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at scala.concurrent.forkjoin.ForkJoinPool.tryAddWorker(ForkJoinPool.java:1672) at scala.concurrent.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1966) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.push(ForkJoinPool.java:1072) at scala.concurrent.forkjoin.ForkJoinTask.fork(ForkJoinTask.java:654) at scala.collection.parallel.ForkJoinTasks$WrappedTask$ This is the error produced by the Spark Driver program which is running on client mode by default so some people say just increase the heap size by passing the --driver-memory 3g flag however the message *"**unable to create new native thread**"* really says that the JVM is asking OS to create a new thread but OS couldn't allocate it anymore and the number of threads a JVM can create by requesting OS is platform dependent but typically it is 32K threads on a 64-bit JVM. so I am wondering why spark is even creating so many threads and how do I control this number?
Spark 1.6 Streaming delay after long run
after upgrade from Spark 1.5 to 1.6(CDH 5.6.0 -> 5.7.1) some of our streaming job getting delay after long run. with a little invesgation, here is what i found. - the same program have no problem with Spark 1.5 - we have two kind of streaming and only those with "updateStateByKey" was affected, - cpu usage getting higher and higher over time ( with 1core@5% at start and 1core@100% after a week ) - data rate is alound 100 event/s, there is no chance for the cpu to work so hard. - process time for a batch delay from 100ms at start to 3s after a week - evening running the same program(for difference input data), not all process delay with the same scale - no warning or error message until it delay too much and went out of memory - process time of customer code seems have no problem - memory/heap usage looks normal to me Im suspecting the problem is comming from updateStateByKey but i cant trace it down any one experience the same problem? -- BR Peter Chan - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
