Hi Selva, The file at /proc/sys/kernel/core_pattern shows:
|/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e This morning, after a clean restart of both Trafodion and Apache OFBiz. the same happened again (hanging OFBiz). This seems to be related to a specific query: SELECT JOB_ID, JOB_NAME, RUN_TIME, POOL_ID, STATUS_ID, PARENT_JOB_ID, PREVIOUS_JOB_ID, SERVICE_NAME, LOADER_NAME, MAX_RETRY, CURRENT_RETRY_COUNT, AUTH_USER_LOGIN_ID, RUN_AS_USER, RUNTIME_DATA_ID, RECURRENCE_INFO_ID, TEMP_EXPR_ID, CURRENT_RECURRENCE_COUNT, MAX_RECURRENCE_COUNT, RUN_BY_INSTANCE_ID, START_DATE_TIME, FINISH_DATE_TIME, CANCEL_DATE_TIME, JOB_RESULT, LAST_UPDATED_STAMP, LAST_UPDATED_TX_STAMP, CREATED_STAMP, CREATED_TX_STAMP FROM OFBIZ.JOB_SANDBOX WHERE (((RUN_TIME <= ? AND START_DATE_TIME IS NULL AND CANCEL_DATE_TIME IS NULL AND RUN_BY_INSTANCE_ID IS NULL) AND (POOL_ID IS NULL OR POOL_ID = ?))) ORDER BY RUN_TIME ASC (The message id: problem_with_server_read) In sqlci I executed a somewhat similar query: select * from ofbiz.job_sandbox This resulted in: # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x00007f71e7a1f907, pid=50530, tid=140127074763584 # # JRE version: Java(TM) SE Runtime Environment (8.0_40-b25) (build 1.8.0_40-b25) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libexecutor.so+0x214907] ExHbaseAccessTcb::createSQRowFromHbaseFormat(long*)+0x3c7 # # Core dump written. Default location: /home/trafodion/core or core.50530 # # An error report file with more information is saved as: # /home/trafodion/hs_err_pid50530.log # # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp The file core.50530 is huge. But the /home/trafodion/hs_err_pid50530.log is better sized to send via email (if you want me too). Best regards, Pierre Smits ORRTIZ.COM <http://www.orrtiz.com> OFBiz based solutions & services OFBiz Extensions Marketplace http://oem.ofbizci.net/oci-2/ On Tue, Mar 29, 2016 at 8:41 PM, Selva Govindarajan < [email protected]> wrote: > Hi Pierre, > > Are you seeing any core dump of the mxosrvr? The core is dumped at the > location based on /proc/sys/kernel/core_pattern. > > Selva > > -----Original Message----- > From: Pierre Smits [mailto:[email protected]] > Sent: Tuesday, March 29, 2016 11:35 AM > To: [email protected] > Subject: Re: problem_with_server_read > > This happened also (approx at the same time): > > *** glibc detected *** mxosrvr: corrupted double-linked list: > 0x0000000002f1f000 *** > > ======= Backtrace: ========= > > /lib64/libc.so.6[0x3e9f275e66] > > /lib64/libc.so.6[0x3e9f279d0f] > > /lib64/libc.so.6(__libc_malloc+0x71)[0x3e9f27a6b1] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x907445)[0x7fe07382b445] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x2dbeb4)[0x7fe0731ffeb4] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x2dcd46)[0x7fe073200d46] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x2dcf76)[0x7fe073200f76] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x8dda98)[0x7fe073801a98] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x497b9f)[0x7fe0733bbb9f] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x49c44a)[0x7fe0733c044a] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x49df4a)[0x7fe0733c1f4a] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x3ebeb8)[0x7fe07330feb8] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x4a843a)[0x7fe0733cc43a] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x4a93e6)[0x7fe0733cd3e6] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0xa5c96f)[0x7fe07398096f] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0xa5ca9c)[0x7fe073980a9c] > > > /usr/jdk64/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so(+0x910ee8)[0x7fe073834ee8] > > /lib64/libpthread.so.0[0x3e9f6079d1] > > /lib64/libc.so.6(clone+0x6d)[0x3e9f2e88fd] > > ======= Memory map: ======== > > 00400000-0051a000 r-xp 00000000 fd:02 4587565 > /home/trafodion/apache-trafodion-1.3.0-incubating-bin/export/bin64/mxosrvr > > 00719000-0071d000 rw-p 00119000 fd:02 4587565 > /home/trafodion/apache-trafodion-1.3.0-incubating-bin/export/bin64/mxosrvr > > 0071d000-00741000 rw-p 00000000 00:00 0 > > 00fc0000-03e32000 rw-p 00000000 00:00 0 > [heap] > > 10000000-14000000 rw-s 00000000 00:04 720913 /SYSV10008cfd (deleted) > > e0000000-e4080000 rw-p 00000000 00:00 0 > > e4080000-f5580000 ---p 00000000 00:00 0 > > f5580000-100000000 rw-p 00000000 00:00 0 > > 100000000-1002a0000 rw-p 00000000 00:00 0 > > 1002a0000-140000000 ---p 00000000 00:00 0 > > 3be7600000-3be7652000 r-xp 00000000 fd:00 2757384 > /usr/lib64/libcurl.so.4.1.1 > > 3be7652000-3be7851000 ---p 00052000 fd:00 2757384 > /usr/lib64/libcurl.so.4.1.1 > > 3be7851000-3be7854000 rw-p 00051000 fd:00 2757384 > /usr/lib64/libcurl.so.4.1.1 > > 3be8600000-3be860e000 r-xp 00000000 fd:00 2359348 > /lib64/liblber-2.4.so.2.10.2 > > 3be860e000-3be880d000 ---p 0000e000 fd:00 2359348 > /lib64/liblber-2.4.so.2.10.2 > > 3be880d000-3be880e000 r--p 0000d000 fd:00 2359348 > /lib64/liblber-2.4.so.2.10.2 > > 3be880e000-3be880f000 rw-p 0000e000 fd:00 2359348 > /lib64/liblber-2.4.so.2.10.2 > > 3be9600000-3be9602000 r-xp 00000000 fd:00 2359333 /lib64/libfreebl3.so > > 3be9602000-3be9801000 ---p 00002000 fd:00 2359333 /lib64/libfreebl3.so > > 3be9801000-3be9802000 r--p 00001000 fd:00 2359333 /lib64/libfreebl3.so > > 3be9802000-3be9803000 rw-p 00002000 fd:00 2359333 /lib64/libfreebl3.so > > 3be9a00000-3be9a07000 r-xp 00000000 fd:00 2359334 /lib64/libcrypt-2.12.so > > 3be9a07000-3be9c07000 ---p 00007000 fd:00 2359334 /lib64/libcrypt-2.12.so > > 3be9c07000-3be9c08000 r--p 00007000 fd:00 2359334 /lib64/libcrypt-2.12.so > > 3be9c08000-3be9c09000 rw-p 00008000 fd:00 2359334 /lib64/libcrypt-2.12.so > > 3be9c09000-3be9c37000 rw-p 00000000 00:00 0 > > 3beda00000-3beda39000 r-xp 00000000 fd:00 2359335 /lib64/libnspr4.so > > 3beda39000-3bedc39000 ---p 00039000 fd:00 2359335 /lib64/libnspr4.so > > 3bedc39000-3bedc3a000 r--p 00039000 fd:00 2359335 /lib64/libnspr4.so > > 3bedc3a000-3bedc3c000 rw-p 0003a000 fd:00 2359335 /lib64/libnspr4.so > > 3bedc3c000-3bedc3e000 rw-p 00000000 00:00 0 > > 3bede00000-3bede25000 r-xp 00000000 fd:00 2758097 /usr/lib64/libnssutil3.so > > 3bede25000-3bee025000 ---p 00025000 fd:00 2758097 /usr/lib64/libnssutil3.so > > 3bee025000-3bee02b000 r--p 00025000 fd:00 2758097 /usr/lib64/libnssutil3.so > > 3bee02b000-3bee02c000 rw-p 0002b000 fd:00 2758097 /usr/lib64/libnssutil3.so > > 3bee200000-3bee204000 r-xp 00000000 fd:00 2359337 /lib64/libplc4.so > > 3bee204000-3bee403000 ---p 00004000 fd:00 2359337 /lib64/libplc4.so > > 3bee403000-3bee404000 r--p 00003000 fd:00 2359337 /lib64/libplc4.so > > 3bee404000-3bee405000 rw-p 00004000 fd:00 2359337 /lib64/libplc4.so > > 3bee600000-3bee603000 r-xp 00000000 fd:00 2359341 /lib64/libplds4.so > > 3bee603000-3bee802000 ---p 00003000 fd:00 2359341 /lib64/libplds4.so > > 3bee802000-3bee803000 r--p 00002000 fd:00 2359341 /lib64/libplds4.so > > 3bee803000-3bee804000 rw-p 00003000 fd:00 2359341 /lib64/libplds4.so > > 3beea00000-3beeb37000 r-xp 00000000 fd:00 2769235 /usr/lib64/libnss3.so > > 3beeb37000-3beed36000 ---p 00137000 fd:00 2769235 /usr/lib64/libnss3.so > > 3beed36000-3beed3b000 r--p 00136000 fd:00 2769235 /usr/lib64/libnss3.so > > 3beed3b000-3beed3d000 rw-p 0013b000 fd:00 2769235 /usr/lib64/libnss3.so > > 3beed3d000-3beed3f000 rw-p 00000000 00:00 0 > > 3beee00000-3beee28000 r-xp 00000000 fd:00 2769244 /usr/lib64/libsmime3.so > > 3beee28000-3bef027000 ---p 00028000 fd:00 2769244 /usr/lib64/libsmime3.so > > 3bef027000-3bef02b000 r--p 00027000 fd:00 2769244 /usr/lib64/libsmime3.so > > 3bef02b000-3bef02c000 rw-p 0002b000 fd:00 2769244 /usr/lib64/libsmime3.so > > 3bef200000-3bef23b000 r-xp 00000000 fd:00 2769242 /usr/lib64/libssl3.so > > 3bef23b000-3bef43b000 ---p 0003b000 fd:00 2769242 /usr/lib64/libssl3.so > > 3bef43b000-3bef43e000 r--p 0003b000 fd:00 2769242 /usr/lib64/libssl3.so > > 3bef43e000-3bef43f000 rw-p 0003e000 fd:00 2769242 /usr/lib64/libssl3.so > > 3bef43f000-3bef440000 rw-p 00000000 00:00 0 > > 3bef600000-3bef619000 r-xp 00000000 fd:00 2779034 > /usr/lib64/libsasl2.so.2.0.23 > > 3bef619000-3bef818000 ---p 00019000 fd:00 2779034 > /usr/lib64/libsasl2.so.2.0.23 > > 3bef818000-3bef819000 r--p 00018000 fd:00 2779034 > /usr/lib64/libsasl2.so.2.0.23 > > 3bef819000-3bef81a000 rw-p 00019000 fd:00 2779034 > /usr/lib64/libsasl2.so.2.0.23 > > 3befa00000-3befa4d000 r-xp 00000000 fd:00 2359366 > /lib64/libldap-2.4.so.2.10.2 > > 3befa4d000-3befc4c000 ---p 0004d000 fd:00 2359366 > /lib64/libldap-2.4.so.2.10.2 > > 3befc4c000-3befc4e000 r--p 0004c000 fd:00 2359366 > /lib64/libldap-2.4.so.2.10.2 > > 3befc4e000-3befc50000 rw-p 0004e000 fd:00 2359366 > /lib64/libldap-2.4.so.2.10.2 > > 3e9ea00000-3e9ea20000 r-xp 00000000 fd:00 2359404 /lib64/ld-2.12.so > > 3e9ec1f000-3e9ec20000 r--p 0001f000 fd:00 2359404 /lib64/ld-2.12.so > > > Best regards, > > Pierre Smits > > ORRTIZ.COM <http://www.orrtiz.com> > OFBiz based solutions & services > > OFBiz Extensions Marketplace > http://oem.ofbizci.net/oci-2/ > > On Tue, Mar 29, 2016 at 8:29 PM, Pierre Smits <[email protected]> > wrote: > > > Hi all, > > > > After a restart of both the Trafodion 1.3 Sandbox and Apache ofbiz I > > get the subject of this posting returned in the OFBiz log. The excerpt > > shows: > > > > > > [java] Caused by: org.trafodion.jdbc.t4.HPT4Exception: The > > message > > id: problem_with_server_read > > > > [java] at > > org.trafodion.jdbc.t4.HPT4Messages.createSQLException(HPT4Messages.jav > > a:304) > > ~[jdbcT4.jar:?] > > > > [java] at > > org.trafodion.jdbc.t4.InputOutput.doIO(InputOutput.java:373) > > ~[jdbcT4.jar:?] > > > > [java] at > > org.trafodion.jdbc.t4.T4Connection.getReadBuffer(T4Connection.java:154 > > ) > > ~[jdbcT4.jar:?] > > > > [java] at > > org.trafodion.jdbc.t4.T4Connection.EndTransaction(T4Connection.java:39 > > 8) > > ~[jdbcT4.jar:?] > > > > [java] at > > org.trafodion.jdbc.t4.InterfaceConnection.endTransaction(InterfaceConn > > ection.java:1222) > > ~[jdbcT4.jar:?] > > > > [java] at > > org.trafodion.jdbc.t4.InterfaceConnection.rollback(InterfaceConnection > > .java:440) > > ~[jdbcT4.jar:?] > > > > [java] at > > org.trafodion.jdbc.t4.TrafT4Connection.rollback(TrafT4Connection.java: > > 1025) > > ~[jdbcT4.jar:?] > > > > > > This continued multiple times until the OFBiz instantiation halts. > > > > Can anybody shed a light what this means and why this is happening? > > > > Best regards, > > > > Pierre Smits > > > > ORRTIZ.COM <http://www.orrtiz.com> > > OFBiz based solutions & services > > > > OFBiz Extensions Marketplace > > http://oem.ofbizci.net/oci-2/ > > >
