One additional possibility (hardcore though) is to attach using gdb to frozen jvm process to see where it hangs. But not sure if it will be possible ie. how much troubled JVM process is.
Check if it hangs always at the same place or at random places (in such case it would be more probably HW problem (memory?)). Of course in such case you can also use debug version of system libs or debug version of JDK/JVM to get more details. But debug JDK is slower. But this require somebody with some gdb / OS experience. Marek ---------- Původní zpráva ---------- Od: Daniel Hobi <d.h...@gmx.ch> Datum: 8. 3. 2013 Předmět: Re: JVM hangs during startup (indexing) (2) "We're already monitoring the JVM process with YourKit. Looking good until the JVM freezes, we get no more stats from the JVM. After the freeze no more connections to the JVM could be made. > It looks suspicious. I would recommend some HW stress test to make sure > there is no HW problem. I am sorry I do not have better idea. Thanks. I'll give that a try. Can you recommend a good stress test tool? Even if the problem is not solved (yet) I want to give you a big thanks for your support! Daniel Am 08.03.2013 um 11:21 schrieb msl...@email.cz: > Last idea is to connect JVM by jconsole and monitor memory sizes if anything > goes wrong. (But any heap area > overflow should not cause this. It would simply switch off something with > warning log or throw OOME with some explanation message.) > > It looks suspicious. I would recommend some HW stress test to make sure > there is no HW problem. I am sorry I do not have better idea. > > > > > > Marek > > > > ---------- Původní zpráva ---------- > Od: Daniel Hobi <d.h...@gmx.ch> > Datum: 8. 3. 2013 > Předmět: Re: JVM hangs during startup (indexing) (2) > > "> Does it happen also on another machine? > > Yes, but only once (of 5…6 reindexing is running) > >> Try latest Oracle JDK to see if it makes any difference > Same effect with latest Oracle JVM (JRE) > > CPU / IO looks ok to me as long as the JVM is running. > As soon as the JVM freezes/hangs CPU/IO is idle > > root@miraculix:~# ulimit -a > core file size (blocks, -c) 0 > data seg size (kbytes, -d) unlimited > scheduling priority (-e) 0 > file size (blocks, -f) unlimited > pending signals (-i) 15774 > max locked memory (kbytes, -l) 64 > max memory size (kbytes, -m) unlimited > open files (-n) 9000 > pipe size (512 bytes, -p) 8 > POSIX message queues (bytes, -q) 819200 > real-time priority (-r) 0 > stack size (kbytes, -s) 8192 > cpu time (seconds, -t) unlimited > max user processes (-u) 15774 > virtual memory (kbytes, -v) unlimited > file locks (-x) unlimited > > Thanks for your input so far! > Daniel > > Am 08.03.2013 um 10:49 schrieb <msl...@email.cz>: > >> It is really weird that JVM process is frozen and you cannot get thread >> dump. I never saw such problem. It looks like HW or OS problem. >> Does it happen also on another machine? From your log I also see you use >> OpenJDK. Try latest Oracle JDK to see if it makes any difference. >> >> >> >> >> If you monitor frozen JVM process is there any activity? (CPU/io). Is > there >> any OS limit on process mem size? Check ulimit settings. (ulimit -a) >> >> >> >> >> Marek >> >> >> "Hi Marek >> >> I use jstack -F <pid> to get thread dumps. >> - Except once the output of jstack is the only one I get (= the first dump > I >> provided) >> - Once (the jvm was not completely frozen, just the indexing made no > further >> progress) jstack also made a dump in our log file (= second dump) >> >> Another result after killing the jvm process with "kill -4 <pid>" (kill - 3 > >> does not work due to a frozen jvm) >> http://pastebin.com/qc2bUkbk >> >> Daniel >> >> Am 08.03.2013 um 10:16 schrieb <msl...@email.cz>: >> >>> I checked 2 thread dumps you provided so I can only comment what I saw >>> there. I did not see any deadlock. Only >>> problem I saw was in main thread where it was waiting for connection. >>> >>> >>> >>> >>> >>> 1. >>> "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x >>> 00007fefd3812000] >>> 2. >>> java.lang.Thread.State: WAITING (on object monitor) >>> 3. >>> at java.lang.Object.$$YJP$$wait(Native Method) >>> 4. >>> at java.lang.Object.wait(Object.java) >>> 5. >>> at java.lang.Object.wait(Object.java:485) >>> 6. >>> at org.apache.commons.pool.impl.GenericObjectPool.borrowObject >>> (GenericObjectPool.java:1104) >>> 7. >>> - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl. >>> GenericObjectPool$Latch) >>> 8. >>> at org.apache.commons.dbcp.PoolingDataSource.getConnection >>> (PoolingDataSource.java:106) >>> 9. >>> at org.apache.commons.dbcp.BasicDataSource.getConnection >>> (BasicDataSource.java:1044) >>> 10. >>> at org.apache.jackrabbit.core.util.db.ConnectionHelper. >>> getConnection(ConnectionHelper.java:439) >>> >>> >>> >>> >>> >>> >>> You should be able to get thread dump any time unless JVM process is >>> completely frozen IMO. Do you use jstack to get thread dump? >>> >>> Marek >>> >>> >>> ---------- Původní zpráva ---------- >>> Od: Daniel Hobi <d.h...@gmx.ch> >>> Datum: 8. 3. 2013 >>> Předmět: Re: JVM hangs during startup (indexing) (2) >>> >>> "Hi Marek >>> >>> Thanks for your suggestions! >>> >>> Unfortunately, changing the JVM gc options so that the JVM will no longer > >>> hang just worked ONCE. >>> Therefore I was not able to get a "nice" stack trace anymore :-( >>> >>> Connection Pool: >>> We had set the max connection pool size to 5. As you suggested I set it > to >> >>> 15. Same effect. Even 50 or 100 did not help. >>> Interestingly, postgres shows just 3 open connection for jackrabbit. Is >> this >>> a suspicious behavior or does it simply not need more connections and is >>> therefore not spawning new connections? >>> >>> Enabling debug, setting login and socket timeout on the postgres-jdbc- >> driver >>> did not help either. >>> I even changed the pool implementation from dbcp to c3p0 in jackrabbit- >> core- >>> 2.6.0 to be "sure" it is not a pool bug. >>> Further we are not able to reproduce the hang with: >>> - Slower machine, postgres 8.3 >>> - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far) >>> - Faster machine, mysql >>> >>> This issue is slowly driving me nuts ;-) >>> >>> Thanks! >>> Daniel >>> >>> >>> >>> Am 07.03.2013 um 16:24 schrieb msl...@email.cz: >>> >>>> Also randomly we had problem with not released connections in JBoss >>>> connection pool and Postgres. Now long time no occurrence. >>>> In any case you can check number/list of open connections in psql . In >> our >>> >>>> case it was >>>> 97 open connection (by default 100 open connections in postgres and 3 > are >> >>>> reserved for admin). It happenned even if we had much smaller >>>> max connection pool size. As it happened randomly we did not find any >>>> solution. We tried to change connection pool size and it does not >>>> happen any more. Still no idea what was cause of this. In any case using > >>>> psql you can find out easily if it is also your problem. >>>> >>>> Query is for 9.x: SELECT * FROM pg_stat_activity; >>>> Marek >>>> >>>> -- >>>> Marek Slama >>>> msl...@email.cz >>>> >>>> >>>> >>>> ---------- Původní zpráva ---------- >>>> Od: msl...@email.cz >>>> Datum: 7. 3. 2013 >>>> Předmět: Re: JVM hangs during startup (indexing) (2) >>>> >>>> "Quick look: It hangs in connection pool - connection pool if it cannot >>>> return free connection to caller it waits forever by default. >>>> How big is your connection pool? (Maximum size? It would be good to > patch >> >>>> connection pool to see who/how many connections is >>>> used and why connections are used not release. In our config we use >>>> connection max pool size 15 and it is enough.) So first >>>> fast fix would be to increase max connection pool size but if there is >>>> problem with not released connection it will not help for long. >>>> >>>> Good news is it is not problem with memory or gc (at least IMO). >>>> >>>> Marek >>>> >>>> -- >>>> Marek Slama >>>> msl...@email.cz >>>> >>>> >>>> >>>> ---------- Původní zpráva ---------- >>>> Od: Daniel Hobi <d.h...@gmx.ch> >>>> Datum: 7. 3. 2013 >>>> Předmět: JVM hangs during startup (indexing) (2) >>>> >>>> "Hi there >>>> >>>> >>>> >>>> Some updates on our issue started here: >>>> >>>> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/% 3 > C >>> 201 >>>> 30115135117.186960%40gmx.net%3E >>>> >>>> >>>> >>>> We're still facing the JVM freeze (possibly during minor gcs) even if... >>>> >>>> ...we updated Jackrabbit from 2.2.13 to 2.6.0. >>>> >>>> ...we updated the Sun JVM to the latest version (1.6.39) and even tried >>>> OpenJDK. >>>> >>>> ...we increased the JVM heap size (up to 1.5gb). >>>> >>>> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) >> available. >>>> >>>> >>>> >>>> But there is good news too: >>>> >>>> - I've just played around with the JVM's garbage collection >>>> options. Even if the index process still hangs the JVM will no longer >>> block. >>>> I was able to get an interesting thread dump: >>>> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8 >>>> >>>> - With the addition JVM Param -XX:MaxNewSize=10m we can make the >>>> hang occur faster (less than 100 minor gcs) >>>> >>>> >>>> >>>> I would be very grateful if someone could have a look at this. We're >>> running >>>> (slowly but surely) out of ideas :-( >>>> >>>> >>>> >>>> Thanks! >>>> >>>> Daniel""""" >"
