We're interested in getting bgq support but we'ré not familiar with cnk at all. So if you know how to manipulate binding there, maybe helping you in adding cnk support would be easier ? Btw, does lstopo work on the compute nodes ? Brice
Jeff Hammond <jhamm...@alcf.anl.gov> a écrit : Hi, I'm interested in seeing hwloc supported on Blue Gene/Q. As it is not listed on http://www.open-mpi.org/projects/hwloc/ nor does it use a standard operating system (although CNK is POSIX and very Linux-like in general), I didn't have a reasonable expectation that it would work without some development, but I verified that some of the tests fail, just to be sure. Fortunately, almost all of the tests passed, except for glibc-sched and hwloc_bind. I suspect this is due to the various incompatibilities between glibc in CNK vs. Linux. The failure of both tests occurs with XLC and GCC, although I report the details below for XLC only. Should I report this as a bug? Is it sufficient to port to Blue Gene/Q if I provide the kernel API calls related to thread location, etc. or would any hwloc developers be interested in Blue Gene/Q access for the purposes of development and testing? Thanks, Jeff [jhammond@cetuslac1 tests]$ ../configure --prefix=/home/jhammond/HWLOC/hwloc-1.4.2rc1/install --enable-static --disable-shared CC=bgxlc_r --host=powerpc64-bgq-linux ==> 16927.output <== ==> 16927.cobaltlog <== /usr/bin/qsub.py -t 15 -n 1 --mode=c1 ./glibc-sched submitted with cwd set to: /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests Command: '/bgsys/drivers/ppcfloor/bin/runjob' '--np' '1' '--ranks-per-node' '1' '--cwd' '/veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests' '--block' 'EAS-20040-31371-128' '--corner' 'R00-M1-N04-J09' '--shape' '1x1x1x1x1' '--envs' 'COBALT_JOBID=16927' '--envs' 'BG_SHAREDMEMSIZE=32' '--verbose' '4' ':' '/veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/./glibc-sched' Info: stdin received from /dev/null Info: stdout sent to /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/16927.output Info: stderr sent to /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/16927.error Job 16927/jhammond/19558: Block EAS-20040-31371-128 for location EAS-31151-31151-1 already booted. Starting task for job 16927. (APG) Info: task completed normally with an exit code of 134; initiating job cleanup and removal ==> 16927.error <== 2012-05-03 14:06:58.888 (INFO ) [0xfffac848a40] EAS-20040-31371-128:1498:ibm.runjob.client.options.Parser: set local socket to runjob_mux from properties file 2012-05-03 14:07:00.892 (INFO ) [0xfffac848a40] EAS-20040-31371-128:34615:ibm.runjob.client.Job: job 34615 started glibc-sched: ../../tests/glibc-sched.c:43: main: Assertion `!err' failed. 2012-05-03 14:07:02.772 (WARN ) [0xfffac848a40] EAS-20040-31371-128:34615:ibm.runjob.client.Job: terminated by signal 6 2012-05-03 14:07:02.772 (WARN ) [0xfffac848a40] EAS-20040-31371-128:34615:ibm.runjob.client.Job: abnormal termination by signal 6 from rank 0 ==> 16912.output <== system set is 0xffffffff,0xffffffff Bind this singlethreaded process : FAILED (3, No such process) ==> 16912.cobaltlog <== /usr/bin/qsub.py -t 15 -n 1 --mode=c1 ./hwloc_bind submitted with cwd set to: /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests Command: '/bgsys/drivers/ppcfloor/bin/runjob' '--np' '1' '--ranks-per-node' '1' '--cwd' '/veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests' '--block' 'EAS-20040-31371-128' '--corner' 'R00-M1-N04-J27' '--shape' '1x1x1x1x1' '--envs' 'COBALT_JOBID=16912' '--envs' 'BG_SHAREDMEMSIZE=32' '--verbose' '4' ':' '/veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/./hwloc_bind' Info: stdin received from /dev/null Info: stdout sent to /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/16912.output Info: stderr sent to /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/16912.error Job 16912/jhammond/19536: Block EAS-20040-31371-128 for location EAS-20141-20141-1 already booted. Starting task for job 16912. (APG) Info: task completed normally with an exit code of 134; initiating job cleanup and removal ==> 16912.error <== 2012-05-03 14:02:09.516 (INFO ) [0xfffb35c8a40] EAS-20040-31371-128:31883:ibm.runjob.client.options.Parser: set local socket to runjob_mux from properties file 2012-05-03 14:02:11.526 (INFO ) [0xfffb35c8a40] EAS-20040-31371-128:34593:ibm.runjob.client.Job: job 34593 started *** glibc detected *** /veas_home/jhammond/HWLOC/hwloc-1.4.2rc1/build/tests/./hwloc_bind: free(): invalid next size (fast): 0x00000019c5016b00 *** ======= Backtrace: ========= [0x103d398] [0x1042ad8] [0x1049800] [0x101e178] [0x101e220] [0x1024520] [0x1024974] [0x1024ccc] [0x1024eb0] [0x1016478] [0x10014a4] [0x1001958] [0x10260d8] [0x10263d4] ======= Memory map: ======== 10000000-10060000 r-xp 00000000 00:0f 1504118 /sbin/sysiod 10060000-10070000 rw-p 00050000 00:0f 1504118 /sbin/sysiod 10070000-10100000 rw-p 00000000 00:00 0 [heap] fff90000000-fff90030000 rw-p 00000000 00:00 0 fff90030000-fff94000000 ---p 00000000 00:00 0 fff94c00000-fff94d00000 rw-p 00000000 00:00 0 fff95a00000-fff95a30000 rw-p 00000000 00:00 0 fff95a30000-fff95a40000 -w-s 3fcc48f0000 00:0e 11206 /dev/infiniband/uverbs0 fff95a40000-fff95a50000 -w-s 3fcc08f0000 00:0e 11206 /dev/infiniband/uverbs0 fff95a50000-fff95a60000 r-xp 00000000 00:0f 3369690 /usr/lib64/libbgvrnic-rdmav2.so fff95a60000-fff95a70000 rw-p 00000000 00:0f 3369690 /usr/lib64/libbgvrnic-rdmav2.so fff95a70000-fff95a80000 ---p 00000000 00:00 0 fff95a80000-fff96480000 rw-p 00000000 00:00 0 fff96480000-fff964b0000 r-xp 00000000 00:0f 3350116 /lib64/libselinux.so.1 fff964b0000-fff964c0000 r--p 00020000 00:0f 3350116 /lib64/libselinux.so.1 fff964c0000-fff964d0000 rw-p 00030000 00:0f 3350116 /lib64/libselinux.so.1 fff964d0000-fff96540000 r-xp 00000000 00:0f 2638150 /lib64/libfreebl3.so fff96540000-fff96550000 r--p 00060000 00:0f 2638150 /lib64/libfreebl3.so fff96550000-fff96560000 rw-p 00070000 00:0f 2638150 /lib64/libfreebl3.so fff96560000-fff96570000 r-xp 00000000 00:0f 2939812 /lib64/libkeyutils.so.1.3 fff96570000-fff96580000 r--p 00000000 00:0f 2939812 /lib64/libkeyutils.so.1.3 fff96580000-fff96590000 rw-p 00010000 00:0f 2939812 /lib64/libkeyutils.so.1.3 fff96590000-fff965a0000 r-xp 00000000 00:0f 2924424 /lib64/libkrb5support.so.0.1 fff965a0000-fff965b0000 r--p 00000000 00:0f 2924424 /lib64/libkrb5support.so.0.1 fff965b0000-fff965c0000 rw-p 00010000 00:0f 2924424 /lib64/libkrb5support.so.0.1 fff965c0000-fff97510000 r-xp 00000000 00:0f 914026 /usr/lib64/libicudata.so.42.1 fff97510000-fff97520000 rw-p 00f40000 00:0f 914026 /usr/lib64/libicudata.so.42.1 fff97520000-fff97550000 r-xp 00000000 00:0f 1277798 /usr/lib64/libsasl2.so.2.0.23 fff97550000-fff97560000 r--p 00020000 00:0f 1277798 /usr/lib64/libsasl2.so.2.0.23 fff97560000-fff97570000 rw-p 00030000 00:0f 1277798 /usr/lib64/libsasl2.so.2.0.23 fff97570000-fff975d0000 r-xp 00000000 00:0f 1327117 /lib64/libnspr4.so fff975d0000-fff975e0000 r--p 00050000 00:0f 1327117 /lib64/libnspr4.so fff975e0000-fff975f0000 rw-p 00060000 00:0f 1327117 /lib64/libnspr4.so fff975f0000-fff97600000 r-xp 00000000 00:0f 1250566 /lib64/libplc4.so fff97600000-fff97610000 r--p 00000000 00:0f 1250566 /lib64/libplc4.so fff97610000-fff97620000 rw-p 00010000 00:0f 1250566 /lib64/libplc4.so fff97620000-fff97630000 r-xp 00000000 00:0f 2447916 /lib64/libplds4.so fff97630000-fff97640000 r--p 00000000 00:0f 2447916 /lib64/libplds4.so fff97640000-fff97650000 rw-p 00010000 00:0f 2447916 /lib64/libplds4.so fff97650000-fff97680000 r-xp 00000000 00:0f 2121934 /usr/lib64/libnssutil3.so fff97680000-fff97690000 r--p 00020000 00:0f 2121934 /usr/lib64/libnssutil3.so fff97690000-fff976a0000 rw-p 00030000 00:0f 2121934 /usr/lib64/libnssutil3.so fff976a0000-fff976b0000 rw-p 00000000 00:00 0 fff976b0000-fff97840000 r-xp 00000000 00:0f 811615 /usr/lib64/libnss3.so fff97840000-fff97850000 r--p 00180000 00:0f 811615 /usr/lib64/libnss3.so fff97850000-fff97860000 rw-p 00190000 00:0f 811615 /usr/lib64/libnss3.so fff97860000-fff97870000 rw-p 00000000 00:00 0 fff97870000-fff978b0000 r-xp 00000000 00:0f 1001793 /usr/lib64/libsmime3.so fff978b0000-fff978c0000 r--p 00030000 00:0f 1001793 /usr/lib64/libsmime3.so fff978c0000-fff978d0000 rw-p 00040000 00:0f 1001793 /usr/lib64/libsmime3.so fff978d0000-fff97920000 r-xp 00000000 00:0f 2769018 /usr/lib64/libssl3.so fff97920000-fff97930000 r--p 00040000 00:0f 2769018 /usr/lib64/libssl3.so fff97930000-fff97940000 rw-p 00050000 00:0f 2769018 /usr/lib64/libssl3.so fff97940000-fff97960000 r-xp 00000000 00:0f 3219112 /lib64/libresolv-2.12.so fff97960000-fff97970000 r--p 00010000 00:0f 3219112 /lib64/libresolv-2.12.so fff97970000-fff97980000 rw-p 00020000 00:0f 3219112 /lib64/libresolv-2.12.so fff97980000-fff97990000 r-xp 00000000 00:0f 822832 /lib64/libcrypt-2.12.so fff97990000-fff979a0000 r--p 00000000 00:0f 822832 /lib64/libcrypt-2.12.so fff979a0000-fff979b0000 rw-p 00010000 00:0f 822832 /lib64/libcrypt-2.12.so fff979b0000-fff979d0000 rw-p 00000000 00:00 0 fff979d0000-fff979e0000 r-xp 00000000 00:0f 1327124 /lib64/libuuid.so.1.3.0 fff979e0000-fff979f0000 rw-p 00000000 00:0f 1327124 /lib64/libuuid.so.1.3.0 fff979f0000-fff97a10000 r-xp 00000000 00:0f 2029697 /lib64/libz.so.1.2.3 fff97a10000-fff97a20000 r--p 00010000 00:0f 2029697 /lib64/libz.so.1.2.3 fff97a20000-fff97a30000 rw-p 00020000 00:0f 2029697 /lib64/libz.so.1.2.3 fff97a30000-fff97c50000 r-xp 00000000 00:0f 2098005 /usr/lib64/libcrypto.so.1.0.0 fff97c50000-fff97c70000 r--p 00210000 00:0f 2098005 /usr/lib64/libcrypto.so.1.0.0 fff97c70000-fff97c90000 rw-p 00230000 00:0f 2098005 /usr/lib64/libcrypto.so.1.0.0 fff97c90000-fff97ca0000 rw-p 00000000 00:00 0 fff97ca0000-fff97ce0000 r-xp 00000000 00:0f 1250555 /lib64/libk5crypto.so.3.1 fff97ce0000-fff97cf0000 r--p 00030000 00:0f 1250555 /lib64/libk5crypto.so.3.1 fff97cf0000-fff97d00000 rw-p 00040000 00:0f 1250555 /lib64/libk5crypto.so.3.1 fff97d00000-fff97d10000 r-xp 00000000 00:0f 1327131 /lib64/libcom_err.so.2.1 fff97d10000-fff97d20000 r--p 00000000 00:0f 1327131 /lib64/libcom_err.so.2.1 fff97d20000-fff97d30000 rw-p 00010000 00:0f 1327131 /lib64/libcom_err.so.2.1 fff97d30000-fff97e40000 r-xp 00000000 00:0f 3350124 /lib64/libkrb5.so.3.3 fff97e40000-fff97e50000 r--p 00100000 00:0f 3350124 /lib64/libkrb5.so.3.3 fff97e50000-fff97e60000 rw-p 00110000 00:0f 3350124 /lib64/libkrb5.so.3.3 fff97e60000-fff97eb0000 r-xp 00000000 00:0f 1327132 /lib64/libgssapi_krb5.so.2.2 fff97eb0000-fff97ec0000 r--p 00040000 00:0f 1327132 /lib64/libgssapi_krb5.so.2.2 fff97ec0000-fff97ed0000 rw-p 00050000 00:0f 1327132 /lib64/libgssapi_krb5.so.2.2 fff97ed0000-fff980f0000 r-xp 00000000 00:0f 159290 /usr/lib64/libicui18n.so.42.1 fff980f0000-fff98120000 rw-p 00210000 00:0f 159290 /usr/lib64/libicui18n.so.42.1 fff98120000-fff982c0000 r-xp 00000000 00:0f 2142969 /usr/lib64/libicuuc.so.42.1 fff982c0000-fff982e0000 rw-p 001a0000 00:0f 2142969 /usr/lib64/libicuuc.so.42.1 fff982e0000-fff982f0000 r-xp 00000000 00:0f 1250562 /lib64/libdl-2.12.so fff982f0000-fff98300000 r--p 00000000 00:0f 1250562 /lib64/libdl-2.12.so fff98300000-fff98310000 rw-p 00010000 00:0f 1250562 /lib64/libdl-2.12.so fff98310000-fff98350000 r-xp 00000000 00:0f 78455 /usr/lib64/libapr-1.so.0.3.9 fff98350000-fff98360000 rw-p 00040000 00:0f 78455 /usr/lib64/libapr-1.so.0.3.9 fff98360000-fff98530000 r-xp 00000000 00:0f 3221811 /lib64/libdb-4.7.so fff98530000-fff98550000 rw-p 001d0000 00:0f 3221811 /lib64/libdb-4.7.so fff98550000-fff98590000 r-xp 00000000 00:0f 3219122 /lib64/libexpat.so.1.5.2 fff98590000-fff985a0000 rw-p 00030000 00:0f 3219122 /lib64/libexpat.so.1.5.2 fff985a0000-fff985c0000 r-xp 00000000 00:0f 2029706 /lib64/liblber-2.4.so.2.5.6 fff985c0000-fff985d0000 r--p 00010000 00:0f 2029706 /lib64/liblber-2.4.so.2.5.6 fff985d0000-fff985e0000 rw-p 00020000 00:0f 2029706 /lib64/liblber-2.4.so.2.5.6 fff985e0000-fff98650000 r-xp 00000000 00:0f 1327127 /lib64/libldap-2.4.so.2.5.6 fff98650000-fff98660000 r--p 00060000 00:0f 1327127 /lib64/libldap-2.4.so.2.5.6 fff98660000-fff98670000 rw-p 00070000 00:0f 1327127 /lib64/libldap-2.4.so.2.5.6 fff98670000-fff986b0000 r-xp 00000000 00:0f 571721 /usr/lib64/libaprutil-1.so.0.3.9 fff986b0000-fff986c0000 rw-p 00030000 00:0f 571721 /usr/lib64/libaprutil-1.so.0.3.9 fff986c0000-fff986d0000 r-xp 00000000 00:0f 2632456 /lib64/librt-2.12.so fff986d0000-fff986e0000 r--p 00000000 00:0f 2632456 /lib64/librt-2.12.so fff986e0000-fff986f0000 rw-p 00010000 00:0f 2632456 /lib64/librt-2.12.so fff986f0000-fff98760000 r-xp 00000000 00:0f 2078098 /usr/lib64/libssl.so.1.0.0 fff98760000-fff98770000 r--p 00060000 00:0f 2078098 /usr/lib64/libssl.so.1.0.0 fff98770000-fff98780000 rw-p 00070000 00:0f 2078098 /usr/lib64/libssl.so.1.0.0 fff98780000-fff988b0000 r-xp 00000000 00:0f 3397951 /usr/lib64/libboost_regex-mt.so.5 fff988b0000-fff988c0000 rw-p 00130000 00:0f 3397951 /usr/lib64/libboost_regex-mt.so.5 fff988c0000-fff988e0000 r-xp 00000000 00:0f 3135862 /usr/lib64/libboost_thread-mt.so.5 fff988e0000-fff988f0000 rw-p 00010000 00:0f 3135862 /usr/lib64/libboost_thread-mt.so.5 fff988f0000-fff98980000 r-xp 00000000 00:0f 784193 /usr/lib64/libboost_serialization-mt.so.5 fff98980000-fff98990000 rw-p 00080000 00:0f 784193 /usr/lib64/libboost_serialization-mt.so.5 fff98990000-fff989a0000 r-xp 00000000 00:0f 2912814 /usr/lib64/libmlx4-rdmav2.so fff989a0000-fff989b0000 rw-p 00000000 00:0f 2912814 /usr/lib64/libmlx4-rdmav2.so fff989b0000-fff989c0000 r-xp 00000000 00:0f 2213992 /usr/lib64/librdmacm.so.1.0.0 fff989c0000-fff989d0000 rw-p 00000000 00:0f 2213992 /usr/lib64/librdmacm.so.1.0.0 fff989d0000-fff989f0000 r-xp 00000000 00:0f 948700 /usr/lib64/libibverbs.so.1.0.0 fff989f0000-fff98a00000 rw-p 00010000 00:0f 948700 /usr/lib64/libibverbs.so.1.0.0 fff98a00000-fff98d60000 r-xp 00000000 00:0f 2989947 /lib/liblog4cxx.so.10.0.0 fff98d60000-fff98db0000 rw-p 00360000 00:0f 2989947 /lib/liblog4cxx.so.10.0.0 fff98db0000-fff98dc0000 rw-p 00000000 00:00 0 fff98dc0000-fff98f80000 r-xp 00000000 00:0f 76138 /lib64/libc-2.12.so fff98f80000-fff98f90000 r--p 001b0000 00:0f 76138 /lib64/libc-2.12.so fff98f90000-fff98fa0000 rw-p 001c0000 00:0f 76138 /lib64/libc-2.12.so fff98fa0000-fff98fb0000 rw-p 00000000 00:00 0 fff98fb0000-fff98fd0000 r-xp 00000000 00:0f 3350121 /lib64/libpthread-2.12.so fff98fd0000-fff98fe0000 r--p 00010000 00:0f 3350121 /lib64/libpthread-2.12.so fff98fe0000-fff98ff0000 rw-p 00020000 00:0f 3350121 /lib64/libpthread-2.12.so fff98ff0000-fff99010000 r-xp 00000000 00:0f 822834 /lib64/libgcc_s-4.4.6-20110824.so.1 fff99010000-fff99020000 rw-p 00010000 00:0f 822834 /lib64/libgcc_s-4.4.6-20110824.so.1 fff99020000-fff99100000 r-xp 00000000 00:0f 3219113 /lib64/libm-2.12.so fff99100000-fff99110000 r--p 000d0000 00:0f 3219113 /lib64/libm-2.12.so fff99110000-fff99120000 rw-p 000e0000 00:0f 3219113 /lib64/libm-2.12.so fff99120000-fff99280000 r-xp 00000000 00:0f 2869896 /usr/lib64/libstdc++.so.6.0.13 fff99280000-fff99290000 r--p 00150000 00:0f 2869896 /usr/lib64/libstdc++.so.6.0.13 fff99290000-fff992a0000 rw-p 00160000 00:0f 2869896 /usr/lib64/libstdc++.so.6.0.13 fff992a0000-fff992c0000 rw-p 00000000 00:00 0 fff992c0000-fff99320000 r-xp 00000000 00:0f 1455532 /lib64/libbgcios.so.1.0.0 fff99320000-fff99330000 rw-p 00060000 00:0f 1455532 /lib64/libbgcios.so.1.0.0 fff99330000-fff99390000 r-xp 00000000 00:0f 2565847 /usr/lib64/libboost_program_options-mt.so.5 fff99390000-fff993a0000 rw-p 00060000 00:0f 2565847 /usr/lib64/libboost_program_options-mt.so.5 fff993a0000-fff993b0000 r-xp 00000000 00:0f 3263171 /usr/lib64/libboost_system-mt.so.5 fff993b0000-fff993c0000 rw-p 00000000 00:0f 3263171 /usr/lib64/libboost_system-mt.so.5 fff993c0000-fff99860000 r-xp 00000000 00:0f 2989944 /lib/libbgutility.so.1.0.0 fff99860000-fff99890000 rw-p 00490000 00:0f 2989944 /lib/libbgutility.so.1.0.0 fff99890000-fff998b0000 rw-p 00000000 00:00 0 fff998b0000-fff998d0000 r-xp 00000000 00:00 0 [vdso] fff998d0000-fff99900000 r-xp 00000000 00:0f 2939827 /lib64/ld-2.12.so fff99900000-fff99910000 r--p 00020000 00:0f 2939827 /lib64/ld-2.12.so fff99910000-fff99920000 rw-p 00030000 00:0f 2939827 /lib64/ld-2.12.so fffca310000-fffca460000 rw-p 00000000 00:00 0 [stack] 2012-05-03 14:02:13.391 (WARN ) [0xfffb35c8a40] EAS-20040-31371-128:34593:ibm.runjob.client.Job: terminated by signal 6 2012-05-03 14:02:13.392 (WARN ) [0xfffb35c8a40] EAS-20040-31371-128:34593:ibm.runjob.client.Job: abnormal termination by signal 6 from rank 0 -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhamm...@alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond (in-progress) https://wiki.alcf.anl.gov/old/index.php/User:Jhammond (deprecated) https://wiki-old.alcf.anl.gov/index.php/User:Jhammond(deprecated) _____________________________________________ hwloc-users mailing list hwloc-us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/hwloc-users
