bash-3.00$ pargs -e core core 'core' of 9412: java -cp . foo envp[0]: HOMEPATH=/MXD-10700 envp[1]: MANPATH=/opt/SUNWspro/man:/usr/local/man:/usr/local/purify:/usr/openwin/ man:/usr/man:/usr/dt/man:/usr/X11R6/man envp[2]: MX_MEMORY_PASSTHRU=true envp[3]: JDL=ON envp[4]: PIXMAPPATH=/MXD-10700/pixmaps/app:/MXD-10700/pixmaps/db envp[5]: HZ=100 envp[6]: LC_MONETARY=en_US.ISO8859-1 envp[7]: DATABASE_DEVELOPMENT=YES envp[8]: SHELL=/bin/csh envp[9]: TERM=xterm envp[10]: TMPDIR=/tmp envp[11]: LD_PRELOAD=libumem.so.1 envp[12]: LC_NUMERIC=en_US.ISO8859-1 envp[13]: VERSION_STRING=10.7.0.0_clark envp[14]: OLDPWD=/MXD-10700/src/ematrix envp[15]: GALAXYVERSION=2.5 envp[16]: JAVAHOME=/usr/local/jdk1.4.2 envp[17]: UMEM_LOGGING=transaction envp[18]: NDEBUG=on envp[19]: ANT_HOME=/usr/local/jakarta-ant-1.5.4 envp[20]: CONFIG_FILE=/home/matrix/ds-10.7.0.0/matrix.config envp[21]: OPENSSL_HOME=/usr/local/openssl-0.9.6g envp[22]: USER=matrixr envp[23]: MX_MAX_PAGE=2400 envp[24]: LD_LIBRARY_PATH=/usr/j2se/jre/lib/sparc/client:/usr/j2se/jre/lib/sparc:/ usr/j2se/jre/../lib/sparc:/usr/lib/lwp:/MXD-10700/lib/solaris4:/MXD-1070 0/bin/solaris4:/usr/local/xerces_2_1_mx3/lib:/usr/local/galaxy-2.3X/lib/ envp[25]: DBVERSION=ORACLE8 envp[26]: ORACLE_SID=WG73 envp[27]: OPENWINHOME=/usr/openwin envp[28]: SOURCE_VAULT=sync://src.matrixone.net:2647/Projects/MatrixOne envp[29]: OPENLDAP_HOME=/usr/local/openldap-2.0.27 envp[30]: OCI73HOME=/export/home/app/oracle/product/7.3.4 envp[31]: WEBSECURITY=/usr/local/websecurity envp[32]: VISIX_APPS=/usr/local/galaxy-2.3X/etc/vhs.apps envp[33]: JAVAARCH=solaris4 envp[34]: GNUHOME=/usr/local/bin envp[35]: ORDB=ON envp[36]: UMEM_DEBUG=default envp[37]: JDOM_HOME=/usr/local/jdom envp[38]: MAIL=/var/mail/matrixr envp[39]: PATH=.:/home/matrix:/MXD-10700/bin/solaris4:/MXD-10700/bin/solaris4:/MXD -10700/tool:/usr/local/instantclient10_1/bin:/usr/local/bin:/usr/java/bi n:/opt/SUNWspro/bin:/usr/local/jdk1.4.2/bin:/usr/local/jdk1.4.2/bin:/usr /local/bin:/usr/local/jakarta-ant-1.5.4/bin:/usr/local/portlet:/home/syn cmgr/synchronicity/latest/syncinc/bin:/usr/local/websecurity:/usr/local/ jdom:/usr/local/openssl-0.9.6g/lib:/usr/local/openldap-2.0.27/bin:/usr/l ocal/xerces_2_1_mx3/bin:/usr/dt/bin:/usr/openwin/bin:/usr/local/galaxy-2 .3X/bin:/bin:/usr/bin:/usr/lib:/usr/ucb:/usr/ccs/bin:/etc:/usr/etc:/usr/ X11R6/bin:/usr/helpviewer/ebt/bin:/usr/local/bin/ebt envp[40]: PORTLET_HOME=/usr/local/portlet envp[41]: OCIHOME=/usr/local/instantclient10_1 envp[42]: WEBVERSION=10.7.0.0 envp[43]: LC_MESSAGES=C envp[44]: COLORPATH=/usr/openwin/lib envp[45]: LC_COLLATE=en_US.ISO8859-1 envp[46]: PWD=/home/matrix envp[47]: JAVA_HOME=/usr/local/jdk1.4.2 envp[48]: EDITOR=/bin/vi envp[49]: MATRIXREVISION=B10-7:Latest envp[50]: J2EE_HOME=/usr/local/j2sdkee1.3 envp[51]: GALAXYHOME=/usr/local/galaxy-2.3X envp[52]: TZ=US/Eastern envp[53]: SHLVL=1 envp[54]: MXP=/MXD-10700 envp[55]: HOME=/home/matrix envp[56]: MATRIXHOME=/MXD-10700 envp[57]: LD_DEBUG=all envp[58]: SYNC_DIR=/home/syncmgr/synchronicity/latest/syncinc envp[59]: XERCES_HOME=/usr/local/xerces_2_1_mx3 envp[60]: MYSQL_DRIVER=YES envp[61]: LOGNAME=matrixr envp[62]: ODBCHOME=/usr/local/unixODBC-2.2.10 envp[63]: WEBSECURITYHOME=/usr/local/websecurity envp[64]: CLASSPATH=/usr/local/j2sdkee1.3/j2ee.jar envp[65]: LC_CTYPE=en_US.ISO8859-1 envp[66]: MATRIXBIN=/MXD-10700/bin/solaris4 envp[67]: GALAXY_LANG=japanese envp[68]: ARCH=solaris4 envp[69]: LD_DEBUG_FILE=ld_debug.txt envp[70]: MXD=/MXD-10700 envp[71]: DISPLAY=cmilliken-dev:0.0 envp[72]: ORACLE_HOME=/usr/local/instantclient10_1 envp[73]: LC_TIME=en_US.ISO8859-1 envp[74]: OCI80HOME=/usr/local/instantclient10_1 envp[75]: _=/usr/java/bin/java envp[76]: NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat envp[77]: XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
-----Original Message----- From: Jonathan Adams [mailto:[email protected]] Sent: Monday, March 06, 2006 7:12 PM To: Milliken, Clark Cc: David McDaniel (damcdani); Rod.Evans at sun.com; tools-linking at opensolaris.org Subject: Re: [tools-linking] shared lib using libCstd under java On Mon, Mar 06, 2006 at 06:49:35PM -0500, Milliken, Clark wrote: > This is from a different machine/build than the one I previously gave > pstack and pmap output from...this environment is a bit more complex as > there are a few more shared libs involved, but it's the same symptom and > it's what I have available to me at the moment. > > I assume that you only want the dump of the address in the free call... Yup; so somehow we are deleting (which calls free()) a totally invalid pointer; I've run your mdb output through c++filt: > > ::umem_status > Status: ready and active > Concurrency: 4 > Logs: transaction=128k > Message buffer: > free(f1fd2860): invalid or corrupted buffer > stack trace: > libumem.so.1'free+0x54 > libCrun.so.1'void operator delete(void*)+0x4 > libCstd_isa.so.1'std::string &std::string::operator=(const std::string &)+0xa8 > libCstd.so.1'std::string *__rwstd::locale_vector<std::string>::resize(unsigned,std::string )+0xbc > libCstd.so.1'__rwstd::locale_imp::locale_imp #Nvariant 1(unsigned,unsigned)+0xb0 > libCstd.so.1'void std::locale::init()+0x40 > libCstd.so.1'std::istream::basic_istream(std::ios_base::EmptyCtor)+0x70 > libCstd.so.1'?? (0xef875020) > libCstd.so.1'?? (0xef876108) > libCstd.so.1'_init+0x1e0 > ld.so.1'?? (0xff3c0254) > ld.so.1'?? (0xff3c56d4) > ld.so.1'?? (0xff3c5818) > ld.so.1'dlopen+0xac > libhpi.so'?? (0xfed52344) > libjvm.so'JVM_LoadLibrary+0xe8 > libjava.so'Java_java_lang_ClassLoader_00024NativeLibrary_load+0xe8 > ?? (0xfac0bbc8) > ?? (0xfac05c64) > ?? (0xfac05a44) > ?? (0xfac05c64) > ?? (0xfac05c64) > ?? (0xfac05c64) > ?? (0xfac00118) > libjvm.so'?? (0xfe8d4b48) > libjvm.so'?? (0xfe8ecaf8) > libjvm.so'?? (0xfe988d3c) > java'main+0x1560 > java'_start+0x108 > > > > f1fd2860::whatis > f1fd2860 is unknown This means that umem has no idea where this pointer comes from. > > f1fd2860-8,20::dump > 0 1 2 3 4 5 6 7 \/ 9 a b c d e f 01234567v9abcdef > f1fd2850: f092aa40 f092c138 f0929ad8 00000001 ... at ...8........ > f1fd2860: 00000000 00000000 00000000 00000000 ................ > f1fd2870: 00000000 00000000 ffffffff 00000000 ................ This doesn't look like a umem buffer at all; the first two bytes are f0929ad8 00000001, where the first is typically a small number, and the second is a large number; the two XOR together to 0x3a10C000. looking up the address in the core file: > pmap of this core: > > core 'core' of 9412: java -cp . foo > 00010000 40K r-x-- /usr/j2se/bin/java > 00028000 8K rwx-- /usr/j2se/bin/java > 0002A000 2200K rwx-- [ heap ] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is where umem-managed storage will be. > EF780000 1376K r-x-- /usr/lib/libCstd.so.1 > EF8E6000 32K rwx-- /usr/lib/libCstd.so.1 > EF8EE000 8K rwx-- /usr/lib/libCstd.so.1 > EF900000 3464K r-x-- > /usr/local/galaxy-2.3X/lib/libvgalaxy-unicode.so.7 > EFC70000 152K rwx-- > /usr/local/galaxy-2.3X/lib/libvgalaxy-unicode.so.7 > EFC96000 80K rwx-- > /usr/local/galaxy-2.3X/lib/libvgalaxy-unicode.so.7 > EFD00000 1976K r-x-- > /home/matrix/xerces/xerces-c-src2_1_0/lib/libxerces-c.so.21.0 > EFEFC000 648K rwx-- > /home/matrix/xerces/xerces-c-src2_1_0/lib/libxerces-c.so.21.0 > EFF9E000 8K rwx-- > /home/matrix/xerces/xerces-c-src2_1_0/lib/libxerces-c.so.21.0 > F0000000 32040K r-x-- > /home/matrix/ds-10.7.0.0/MXD-10700/bin/solaris4/libeMatrix.so > F1F58000 960K rwx-- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ this is where the buffer is. > F2048000 24K rwx-- > F2100000 560K r-x-- /usr/openwin/lib/libX11.so.4 > F219C000 24K rwx-- /usr/openwin/lib/libX11.so.4 ... So it looks like someone is deleting a random pointer. Could you send the output of 'pargs -e core' as well? Cheers, - jonathan > Thanks, > > Clark > > > -----Original Message----- > From: Jonathan Adams [mailto:jonathan.adams at sun.com] > Sent: Monday, March 06, 2006 6:14 PM > To: Milliken, Clark > Cc: David McDaniel (damcdani); Rod.Evans at sun.com; > tools-linking at opensolaris.org > Subject: Re: [tools-linking] shared lib using libCstd under java > > On Mon, Mar 06, 2006 at 06:06:36PM -0500, Milliken, Clark wrote: > > Optimized builds crash consistently with or without libumem. Debug > > builds only crash with libumem. It's the same stack for both... > > Alright; what is the output of: > > % mdb core > ... > > ::umem_status > > > Also, look at the ::umem_status output; take the hex number between the > parenthesis, and do: > > > number::whatis > > number-8,20::dump > > With the output from all of that, I can tell you why umem is barfing. > > Cheers, > - jonathan > > > -----Original Message----- > > From: Jonathan Adams [mailto:jonathan.adams at sun.com] > > Sent: Monday, March 06, 2006 6:05 PM > > To: David McDaniel (damcdani) > > Cc: Rod.Evans at sun.com; Milliken, Clark; tools-linking at opensolaris.org > > Subject: Re: [tools-linking] shared lib using libCstd under java > > > > On Mon, Mar 06, 2006 at 02:55:13PM -0800, David McDaniel (damcdani) > > wrote: > > > IIRC, the original traceback showed the abort due to umem > determining > > a > > > deallocation was a duplicate or invalid buffer. Preloading libCstd > > would > > > result in umem being usurped by libc, would it not? And thus the > > > presumed double free isnt caught. Maybe the umem noabort flag might > be > > > worth trying, without the libCstd preload. > > > > The umem noabort flag should only be used as a last resort; there is > > still > > an unresolved problem here. > > > > Does the program crash without libumem? > > > > Cheers, > > - jonathan > > > > > > -----Original Message----- > > > > From: tools-linking-bounces at opensolaris.org > > > > [mailto:tools-linking-bounces at opensolaris.org] On Behalf Of Rod > > Evans > > > > Sent: Monday, March 06, 2006 4:33 PM > > > > To: Milliken, Clark > > > > Cc: tools-linking at opensolaris.org > > > > Subject: Re: [tools-linking] shared lib using libCstd under java > > > > > > > > Milliken, Clark wrote: > > > > > Removing the -Bsymbolic option has no effect. > > > > > > > > > > I created a test prog, that just does just does a: > > > > > > > > > > dlopen(...,RTLD_NOW | RTLD_GLOBAL) and that works fine. > > > > > > > > > > Another tid-bit... > > > > > > > > > > If I LD_PRELOAD=libCstd.so.1, my crash goes away. > > > > > > > > You don't happen to have two different versions of libCstd > > > > being loaded in the process do you? What does a pmap of the > > > > core file reveal? > > > > > > > > If not, then one of the C++/iostreams experts on this alias > > > > will hopefully provide more information. > > > > > > > > -- > > > > Rod > > > > _______________________________________________ > > > > tools-linking mailing list > > > > tools-linking at opensolaris.org > > > > > > > _______________________________________________ > > > tools-linking mailing list > > > tools-linking at opensolaris.org > > > > -- > > Jonathan Adams, Solaris Kernel Development > > -- > Jonathan Adams, Solaris Kernel Development -- Jonathan Adams, Solaris Kernel Development
