It is really weird that JVM process is frozen and you cannot get thread dump. I never saw such problem. It looks like HW or OS problem. Does it happen also on another machine? From your log I also see you use OpenJDK. Try latest Oracle JDK to see if it makes any difference.
If you monitor frozen JVM process is there any activity? (CPU/io). Is there any OS limit on process mem size? Check ulimit settings. (ulimit -a) Marek "Hi Marek I use jstack -F <pid> to get thread dumps. - Except once the output of jstack is the only one I get (= the first dump I provided) - Once (the jvm was not completely frozen, just the indexing made no further progress) jstack also made a dump in our log file (= second dump) Another result after killing the jvm process with "kill -4 <pid>" (kill -3 does not work due to a frozen jvm) http://pastebin.com/qc2bUkbk Daniel Am 08.03.2013 um 10:16 schrieb <msl...@email.cz>: > I checked 2 thread dumps you provided so I can only comment what I saw > there. I did not see any deadlock. Only > problem I saw was in main thread where it was waiting for connection. > > > > > > 1. > "main" prio=10 tid=0x0000000040feb000 nid=0x4fb0 in Object.wait() [0x > 00007fefd3812000] > 2. > java.lang.Thread.State: WAITING (on object monitor) > 3. > at java.lang.Object.$$YJP$$wait(Native Method) > 4. > at java.lang.Object.wait(Object.java) > 5. > at java.lang.Object.wait(Object.java:485) > 6. > at org.apache.commons.pool.impl.GenericObjectPool.borrowObject > (GenericObjectPool.java:1104) > 7. > - locked <0x00000000d9b90c40> (a org.apache.commons.pool.impl. > GenericObjectPool$Latch) > 8. > at org.apache.commons.dbcp.PoolingDataSource.getConnection > (PoolingDataSource.java:106) > 9. > at org.apache.commons.dbcp.BasicDataSource.getConnection > (BasicDataSource.java:1044) > 10. > at org.apache.jackrabbit.core.util.db.ConnectionHelper. > getConnection(ConnectionHelper.java:439) > > > > > > > You should be able to get thread dump any time unless JVM process is > completely frozen IMO. Do you use jstack to get thread dump? > > Marek > > > ---------- Původní zpráva ---------- > Od: Daniel Hobi <d.h...@gmx.ch> > Datum: 8. 3. 2013 > Předmět: Re: JVM hangs during startup (indexing) (2) > > "Hi Marek > > Thanks for your suggestions! > > Unfortunately, changing the JVM gc options so that the JVM will no longer > hang just worked ONCE. > Therefore I was not able to get a "nice" stack trace anymore :-( > > Connection Pool: > We had set the max connection pool size to 5. As you suggested I set it to > 15. Same effect. Even 50 or 100 did not help. > Interestingly, postgres shows just 3 open connection for jackrabbit. Is this > a suspicious behavior or does it simply not need more connections and is > therefore not spawning new connections? > > Enabling debug, setting login and socket timeout on the postgres-jdbc- driver > did not help either. > I even changed the pool implementation from dbcp to c3p0 in jackrabbit- core- > 2.6.0 to be "sure" it is not a pool bug. > Further we are not able to reproduce the hang with: > - Slower machine, postgres 8.3 > - Very fast machine, postgres 8.4 (only hanged 1 of 5 times so far) > - Faster machine, mysql > > This issue is slowly driving me nuts ;-) > > Thanks! > Daniel > > > > Am 07.03.2013 um 16:24 schrieb msl...@email.cz: > >> Also randomly we had problem with not released connections in JBoss >> connection pool and Postgres. Now long time no occurrence. >> In any case you can check number/list of open connections in psql . In our > >> case it was >> 97 open connection (by default 100 open connections in postgres and 3 are >> reserved for admin). It happenned even if we had much smaller >> max connection pool size. As it happened randomly we did not find any >> solution. We tried to change connection pool size and it does not >> happen any more. Still no idea what was cause of this. In any case using >> psql you can find out easily if it is also your problem. >> >> Query is for 9.x: SELECT * FROM pg_stat_activity; >> Marek >> >> -- >> Marek Slama >> msl...@email.cz >> >> >> >> ---------- Původní zpráva ---------- >> Od: msl...@email.cz >> Datum: 7. 3. 2013 >> Předmět: Re: JVM hangs during startup (indexing) (2) >> >> "Quick look: It hangs in connection pool - connection pool if it cannot >> return free connection to caller it waits forever by default. >> How big is your connection pool? (Maximum size? It would be good to patch >> connection pool to see who/how many connections is >> used and why connections are used not release. In our config we use >> connection max pool size 15 and it is enough.) So first >> fast fix would be to increase max connection pool size but if there is >> problem with not released connection it will not help for long. >> >> Good news is it is not problem with memory or gc (at least IMO). >> >> Marek >> >> -- >> Marek Slama >> msl...@email.cz >> >> >> >> ---------- Původní zpráva ---------- >> Od: Daniel Hobi <d.h...@gmx.ch> >> Datum: 7. 3. 2013 >> Předmět: JVM hangs during startup (indexing) (2) >> >> "Hi there >> >> >> >> Some updates on our issue started here: >> >> http://mail-archives.apache.org/mod_mbox/jackrabbit-users/201301.mbox/%3C > 201 >> 30115135117.186960%40gmx.net%3E >> >> >> >> We're still facing the JVM freeze (possibly during minor gcs) even if... >> >> ...we updated Jackrabbit from 2.2.13 to 2.6.0. >> >> ...we updated the Sun JVM to the latest version (1.6.39) and even tried >> OpenJDK. >> >> ...we increased the JVM heap size (up to 1.5gb). >> >> ...we upgraded PostgreSQL to the latest version (8.4.x / 9.2.x) available. >> >> >> >> But there is good news too: >> >> - I've just played around with the JVM's garbage collection >> options. Even if the index process still hangs the JVM will no longer > block. >> I was able to get an interesting thread dump: >> <http://pastebin.com/uSrXpej8> http://pastebin.com/uSrXpej8 >> >> - With the addition JVM Param -XX:MaxNewSize=10m we can make the >> hang occur faster (less than 100 minor gcs) >> >> >> >> I would be very grateful if someone could have a look at this. We're > running >> (slowly but surely) out of ideas :-( >> >> >> >> Thanks! >> >> Daniel""""
