Siegmar, how did you configure openmpi ? which java version did you use ?
i just found a regression and you currently have to explicitly add CFLAGS=-D_REENTRANT CPPFLAGS=-D_REENTRANT to your configure command line if you want to debug this issue (i cannot reproduce it on a solaris 11 x86 virtual machine) you can apply the attached patch, and make sure you configure with --enable-debug and run OMPI_ATTACH=1 mpiexec -n 1 java InitFinalizeMain then you will need to attach the *java* process with gdb, set the _dbg local variable to zero and continue you should get a clean stack trace and hopefully we will be able to help Cheers, Gilles On 2014/10/24 0:03, Siegmar Gross wrote: > Hello Oscar, > > do you have time to look into my problem? Probably Takahiro has a > point and gdb behaves differently on Solaris and Linux, so that > the differing outputs have no meaning. I tried to debug my Java > program, but without success so far, because I wasn't able to get > into the Java program to set a breakpoint or to see the code. Have > you succeeded to debug a mpiJava program? If so, how must I call > gdb (I normally use "gdb mipexec" and then "run -np 1 java ...")? > What can I do to get helpful information to track the error down? > I have attached the error log file. Perhaps you can see if something > is going wrong with the Java interface. Thank you very much for your > help and any hints for the usage of gdb with mpiJava in advance. > Please let me know if I can provide anything else. > > > Kind regards > > Siegmar > > >>> I think that it must have to do with MPI, because everything >>> works fine on Linux and my Java program works fine with an older >>> MPI version (openmpi-1.8.2a1r31804) as well. >> Yes. I also think it must have to do with MPI. >> But java process side, not mpiexec process side. >> >> When you run Java MPI program via mpiexec, a mpiexec process >> process launch a java process. When the java process (your >> Java program) calls a MPI method, native part (written in C/C++) >> of the MPI library is called. It runs in java process, not in >> mpiexec process. I suspect that part. >> >>> On Solaris things are different. >> Are you saying the following difference? >> After this line, >>> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT); >> Linux shows >>> orte_job_state_to_str (state=1) >>> at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217 >>> 217 switch(state) { >> but Solaris shows >>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122 >>> 122 if (NULL == name) { >> Each macro is defined as: >> >> #define ORTE_ACTIVATE_JOB_STATE(j, s) \ >> do { \ >> orte_job_t *shadow=(j); \ >> opal_output_verbose(1, orte_state_base_framework.framework_output, \ >> "%s ACTIVATE JOB %s STATE %s AT %s:%d", \ >> ORTE_NAME_PRINT(ORTE_PROC_MY_NAME), \ >> (NULL == shadow) ? "NULL" : \ >> ORTE_JOBID_PRINT(shadow->jobid), \ >> orte_job_state_to_str((s)), \ >> __FILE__, __LINE__); \ >> orte_state.activate_job_state(shadow, (s)); \ >> } while(0); >> >> #define ORTE_NAME_PRINT(n) \ >> orte_util_print_name_args(n) >> >> #define ORTE_JOBID_PRINT(n) \ >> orte_util_print_jobids(n) >> >> I'm not sure, but I think the gdb on Solaris steps into >> orte_util_print_name_args, but gdb on Linux doesn't step into >> orte_util_print_name_args and orte_util_print_jobids for some >> reason, or orte_job_state_to_str is evaluated before them. >> >> So I think it's not an important difference. >> >> You showed the following lines. >>>>> orterun (argc=5, argv=0xffffffff7fffe0d8) >>>>> at > ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084 >>>>> 1084 while (orte_event_base_active) { >>>>> (gdb) >>>>> 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE); >>>>> (gdb) >> I'm not familiar with this code but I think this part (in mpiexec >> process) is only waiting the java process to terminate (normally >> or abnormally). So I think the problem is not in a mpiexec process >> but in a java process. >> >> Regards, >> Takahiro >> >>> Hi Takahiro, >>> >>>> mpiexec and java run as distinct processes. Your JRE message >>>> says java process raises SEGV. So you should trace the java >>>> process, not the mpiexec process. And more, your JRE message >>>> says the crash happened outside the Java Virtual Machine in >>>> native code. So usual Java program debugger is useless. >>>> You should trace native code part of the java process. >>>> Unfortunately I don't know how to debug such one. >>> I think that it must have to do with MPI, because everything >>> works fine on Linux and my Java program works fine with an older >>> MPI version (openmpi-1.8.2a1r31804) as well. >>> >>> linpc1 x 112 mpiexec -np 1 java InitFinalizeMain >>> Hello! >>> linpc1 x 113 >>> >>> Therefore I single stepped through the program on Linux as well >>> and found a difference launching the process. On Linux I get the >>> following sequence. >>> >>> Breakpoint 1, rsh_launch (jdata=0x614aa0) >>> at > ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876 >>> 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) { >>> (gdb) s >>> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT); >>> (gdb) s >>> orte_job_state_to_str (state=1) >>> at ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217 >>> 217 switch(state) { >>> (gdb) >>> 221 return "PENDING INIT"; >>> (gdb) >>> 317 } >>> (gdb) >>> orte_util_print_jobids (job=4294967295) >>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170 >>> 170 ptr = get_print_name_buffer(); >>> (gdb) >>> >>> >>> >>> On Solaris things are different. >>> >>> Breakpoint 1, rsh_launch (jdata=0x100125250) >>> at > ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876 >>> 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) { >>> (gdb) s >>> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT); >>> (gdb) s >>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122 >>> 122 if (NULL == name) { >>> (gdb) >>> 142 job = orte_util_print_jobids(name->jobid); >>> (gdb) >>> orte_util_print_jobids (job=2673410048) >>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170 >>> 170 ptr = get_print_name_buffer(); >>> (gdb) >>> >>> >>> >>> Is this normal or is it the reason for the crash on Solaris? >>> >>> >>> Kind regards >>> >>> Siegmar >>> >>> >>> >>> >>> >>> >>> >>>> The log file output by JRE may help you. >>>>> # An error report file with more information is saved as: >>>>> # > /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid13080.log >>>> Regards, >>>> Takahiro >>>> >>>>> Hi, >>>>> >>>>> I installed openmpi-dev-124-g91e9686 on Solaris 10 Sparc with >>>>> gcc-4.9.1 to track down the error with my small Java program. >>>>> I started single stepping in orterun.c at line 1081 and >>>>> continued until I got the segmentation fault. I get >>>>> "jdata = 0x0" in version openmpi-1.8.2a1r31804, which is the >>>>> last one which works with Java in my environment, while I get >>>>> "jdata = 0x100125250" in this version. Unfortunately I don't >>>>> know which files or variables are important to look at. Perhaps >>>>> somebody can look at the following lines of code and tell me, >>>>> which information I should provide to solve the problem. I know >>>>> that Solaris isn't any longer on your list of supported systems, >>>>> but perhaps we can get it working again, if you tell me what >>>>> you need and I do the debugging. >>>>> >>>>> /usr/local/gdb-7.6.1_64_gcc/bin/gdb mpiexec >>>>> GNU gdb (GDB) 7.6.1 >>>>> ... >>>>> (gdb) run -np 1 java InitFinalizeMain >>>>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec \ >>>>> -np 1 java InitFinalizeMain >>>>> [Thread debugging using libthread_db enabled] >>>>> [New Thread 1 (LWP 1)] >>>>> [New LWP 2 ] >>>>> # >>>>> # A fatal error has been detected by the Java Runtime Environment: >>>>> # >>>>> # SIGSEGV (0xb) at pc=0xffffffff7ea3c7f0, pid=13064, tid=2 >>>>> ... >>>>> [LWP 2 exited] >>>>> [New Thread 2 ] >>>>> [Switching to Thread 1 (LWP 1)] >>>>> sol_thread_fetch_registers: td_ta_map_id2thr: no thread can be >>>>> found to satisfy query >>>>> (gdb) thread 1 >>>>> [Switching to thread 1 (LWP 1 )] >>>>> #0 0xffffffff7f6173d0 in rtld_db_dlactivity () from > /usr/lib/sparcv9/ld.so.1 >>>>> (gdb) b orterun.c:1081 >>>>> Breakpoint 1 at 0x1000070dc: file >>>>> ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c, line > 1081. >>>>> (gdb) r >>>>> The program being debugged has been started already. >>>>> Start it from the beginning? (y or n) y >>>>> >>>>> Starting program: /usr/local/openmpi-1.9.0_64_gcc/bin/mpiexec -np 1 java >>>>> InitFinalizeMain >>>>> [Thread debugging using libthread_db enabled] >>>>> [New Thread 1 (LWP 1)] >>>>> [New LWP 2 ] >>>>> [Switching to Thread 1 (LWP 1)] >>>>> >>>>> Breakpoint 1, orterun (argc=5, argv=0xffffffff7fffe0d8) >>>>> at > ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1081 >>>>> 1081 rc = orte_plm.spawn(jdata); >>>>> (gdb) print jdata >>>>> $1 = (orte_job_t *) 0x100125250 >>>>> (gdb) s >>>>> rsh_launch (jdata=0x100125250) >>>>> at >>>>> > ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:876 >>>>> 876 if (ORTE_FLAG_TEST(jdata, ORTE_JOB_FLAG_RESTART)) { >>>>> (gdb) s >>>>> 881 ORTE_ACTIVATE_JOB_STATE(jdata, ORTE_JOB_STATE_INIT); >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122 >>>>> 122 if (NULL == name) { >>>>> (gdb) >>>>> 142 job = orte_util_print_jobids(name->jobid); >>>>> (gdb) >>>>> orte_util_print_jobids (job=2502885376) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170 >>>>> 170 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd990) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_jobids (job=2502885376) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172 >>>>> 172 if (NULL == ptr) { >>>>> (gdb) >>>>> 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 182 if (ORTE_JOBID_INVALID == job) { >>>>> (gdb) >>>>> 184 } else if (ORTE_JOBID_WILDCARD == job) { >>>>> (gdb) >>>>> 187 tmp1 = ORTE_JOB_FAMILY((unsigned long)job); >>>>> (gdb) >>>>> 188 tmp2 = ORTE_LOCAL_JOBID((unsigned long)job); >>>>> (gdb) >>>>> 189 snprintf(ptr->buffers[ptr->cntr++], >>>>> (gdb) >>>>> 193 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 194 } >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:143 >>>>> 143 vpid = orte_util_print_vpids(name->vpid); >>>>> (gdb) >>>>> orte_util_print_vpids (vpid=0) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:260 >>>>> 260 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd9a0) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_vpids (vpid=0) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:262 >>>>> 262 if (NULL == ptr) { >>>>> (gdb) >>>>> 268 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 272 if (ORTE_VPID_INVALID == vpid) { >>>>> (gdb) >>>>> 274 } else if (ORTE_VPID_WILDCARD == vpid) { >>>>> (gdb) >>>>> 277 snprintf(ptr->buffers[ptr->cntr++], >>>>> (gdb) >>>>> 281 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 282 } >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:146 >>>>> 146 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffda60) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:148 >>>>> 148 if (NULL == ptr) { >>>>> (gdb) >>>>> 154 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 158 snprintf(ptr->buffers[ptr->cntr++], >>>>> (gdb) >>>>> 162 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 163 } >>>>> (gdb) >>>>> orte_util_print_jobids (job=4294967295) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170 >>>>> 170 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffda60) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_jobids (job=4294967295) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172 >>>>> 172 if (NULL == ptr) { >>>>> (gdb) >>>>> 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 182 if (ORTE_JOBID_INVALID == job) { >>>>> (gdb) >>>>> 183 snprintf(ptr->buffers[ptr->cntr++], >>>>> ORTE_PRINT_NAME_ARGS_MAX_SIZE, "[INVALID]"); >>>>> (gdb) >>>>> 193 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 194 } >>>>> (gdb) >>>>> orte_job_state_to_str (state=1) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217 >>>>> 217 switch(state) { >>>>> (gdb) >>>>> 221 return "PENDING INIT"; >>>>> (gdb) >>>>> 317 } >>>>> (gdb) >>>>> opal_output_verbose (level=1, output_id=0, >>>>> format=0xffffffff7f14dd98 <orte_job_states> >>>>> "\336\257\276\355\336\257\276\355") >>>>> at ../../../openmpi-dev-124-g91e9686/opal/util/output.c:373 >>>>> 373 va_start(arglist, format); >>>>> (gdb) >>>>> 369 { >>>>> (gdb) >>>>> 370 if (output_id >= 0 && output_id < OPAL_OUTPUT_MAX_STREAMS && >>>>> (gdb) >>>>> 377 } >>>>> (gdb) >>>>> orte_state_base_activate_job_state (jdata=0x100125250, state=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:33 >>>>> 33 opal_list_item_t *itm, *any=NULL, *error=NULL; >>>>> (gdb) >>>>> 37 for (itm = opal_list_get_first(&orte_job_states); >>>>> (gdb) >>>>> opal_list_get_first (list=0xffffffff7f14dd98 <orte_job_states>) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_list.h:320 >>>>> 320 opal_list_item_t* item = >>>>> (opal_list_item_t*)list->opal_list_sentinel.opal_list_next; >>>>> (gdb) >>>>> 324 assert(1 == item->opal_list_item_refcount); >>>>> (gdb) >>>>> 325 assert( list == item->opal_list_item_belong_to ); >>>>> (gdb) >>>>> 328 return item; >>>>> (gdb) >>>>> 329 } >>>>> (gdb) >>>>> orte_state_base_activate_job_state (jdata=0x100125250, state=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:38 >>>>> 38 itm != opal_list_get_end(&orte_job_states); >>>>> (gdb) >>>>> opal_list_get_end (list=0xffffffff7f14dd98 <orte_job_states>) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_list.h:399 >>>>> 399 return &(list->opal_list_sentinel); >>>>> (gdb) >>>>> 400 } >>>>> (gdb) >>>>> orte_state_base_activate_job_state (jdata=0x100125250, state=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:37 >>>>> 37 for (itm = opal_list_get_first(&orte_job_states); >>>>> (gdb) >>>>> 40 s = (orte_state_t*)itm; >>>>> (gdb) >>>>> 41 if (s->job_state == ORTE_JOB_STATE_ANY) { >>>>> (gdb) >>>>> 45 if (s->job_state == ORTE_JOB_STATE_ERROR) { >>>>> (gdb) >>>>> 48 if (s->job_state == state) { >>>>> (gdb) >>>>> 49 OPAL_OUTPUT_VERBOSE((1, >>>>> orte_state_base_framework.framework_output, >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:122 >>>>> 122 if (NULL == name) { >>>>> (gdb) >>>>> 142 job = orte_util_print_jobids(name->jobid); >>>>> (gdb) >>>>> orte_util_print_jobids (job=2502885376) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170 >>>>> 170 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd880) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_jobids (job=2502885376) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172 >>>>> 172 if (NULL == ptr) { >>>>> (gdb) >>>>> 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 182 if (ORTE_JOBID_INVALID == job) { >>>>> (gdb) >>>>> 184 } else if (ORTE_JOBID_WILDCARD == job) { >>>>> (gdb) >>>>> 187 tmp1 = ORTE_JOB_FAMILY((unsigned long)job); >>>>> (gdb) >>>>> 188 tmp2 = ORTE_LOCAL_JOBID((unsigned long)job); >>>>> (gdb) >>>>> 189 snprintf(ptr->buffers[ptr->cntr++], >>>>> (gdb) >>>>> 193 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 194 } >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:143 >>>>> 143 vpid = orte_util_print_vpids(name->vpid); >>>>> (gdb) >>>>> orte_util_print_vpids (vpid=0) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:260 >>>>> 260 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd890) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_vpids (vpid=0) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:262 >>>>> 262 if (NULL == ptr) { >>>>> (gdb) >>>>> 268 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 272 if (ORTE_VPID_INVALID == vpid) { >>>>> (gdb) >>>>> 274 } else if (ORTE_VPID_WILDCARD == vpid) { >>>>> (gdb) >>>>> 277 snprintf(ptr->buffers[ptr->cntr++], >>>>> (gdb) >>>>> 281 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 282 } >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:146 >>>>> 146 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd950) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_name_args (name=0x100118380 <orte_process_info+104>) >>>>> at ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:148 >>>>> 148 if (NULL == ptr) { >>>>> (gdb) >>>>> 154 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 158 snprintf(ptr->buffers[ptr->cntr++], >>>>> (gdb) >>>>> 162 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 163 } >>>>> (gdb) >>>>> orte_util_print_jobids (job=4294967295) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:170 >>>>> 170 ptr = get_print_name_buffer(); >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:92 >>>>> 92 if (!fns_init) { >>>>> (gdb) >>>>> 101 ret = opal_tsd_getspecific(print_args_tsd_key, > (void**)&ptr); >>>>> (gdb) >>>>> opal_tsd_getspecific (key=1, valuep=0xffffffff7fffd950) >>>>> at ../../openmpi-dev-124-g91e9686/opal/threads/tsd.h:163 >>>>> 163 *valuep = pthread_getspecific(key); >>>>> (gdb) >>>>> 164 return OPAL_SUCCESS; >>>>> (gdb) >>>>> 165 } >>>>> (gdb) >>>>> get_print_name_buffer () at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:102 >>>>> 102 if (OPAL_SUCCESS != ret) return NULL; >>>>> (gdb) >>>>> 104 if (NULL == ptr) { >>>>> (gdb) >>>>> 113 return (orte_print_args_buffers_t*) ptr; >>>>> (gdb) >>>>> 114 } >>>>> (gdb) >>>>> orte_util_print_jobids (job=4294967295) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/name_fns.c:172 >>>>> 172 if (NULL == ptr) { >>>>> (gdb) >>>>> 178 if (ORTE_PRINT_NAME_ARG_NUM_BUFS == ptr->cntr) { >>>>> (gdb) >>>>> 182 if (ORTE_JOBID_INVALID == job) { >>>>> (gdb) >>>>> 183 snprintf(ptr->buffers[ptr->cntr++], >>>>> ORTE_PRINT_NAME_ARGS_MAX_SIZE, "[INVALID]"); >>>>> (gdb) >>>>> 193 return ptr->buffers[ptr->cntr-1]; >>>>> (gdb) >>>>> 194 } >>>>> (gdb) >>>>> orte_job_state_to_str (state=1) at >>>>> ../../openmpi-dev-124-g91e9686/orte/util/error_strings.c:217 >>>>> 217 switch(state) { >>>>> (gdb) >>>>> 221 return "PENDING INIT"; >>>>> (gdb) >>>>> 317 } >>>>> (gdb) >>>>> opal_output_verbose (level=1, output_id=-1, format=0x1 <Address 0x1 out > of >>>>> bounds>) >>>>> at ../../../openmpi-dev-124-g91e9686/opal/util/output.c:373 >>>>> 373 va_start(arglist, format); >>>>> (gdb) >>>>> 369 { >>>>> (gdb) >>>>> 370 if (output_id >= 0 && output_id < OPAL_OUTPUT_MAX_STREAMS && >>>>> (gdb) >>>>> 377 } >>>>> (gdb) >>>>> orte_state_base_activate_job_state (jdata=0x100125250, state=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:54 >>>>> 54 if (NULL == s->cbfunc) { >>>>> (gdb) >>>>> 62 caddy = OBJ_NEW(orte_state_caddy_t); >>>>> (gdb) >>>>> opal_obj_new_debug (type=0xffffffff7f14c7d8 <orte_state_caddy_t_class>, >>>>> file=0xffffffff7f034c08 >>>>> > "../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c", >>>>> line=62) at > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:249 >>>>> 249 opal_object_t* object = opal_obj_new(type); >>>>> (gdb) >>>>> opal_obj_new (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:465 >>>>> 465 assert(cls->cls_sizeof >= sizeof(opal_object_t)); >>>>> (gdb) >>>>> 470 object = (opal_object_t *) malloc(cls->cls_sizeof); >>>>> (gdb) >>>>> 472 if (0 == cls->cls_initialized) { >>>>> (gdb) >>>>> 473 opal_class_initialize(cls); >>>>> (gdb) >>>>> opal_class_initialize (cls=0xffffffff7f14c7d8 > <orte_state_caddy_t_class>) >>>>> at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:79 >>>>> 79 assert(cls); >>>>> (gdb) >>>>> 84 if (1 == cls->cls_initialized) { >>>>> (gdb) >>>>> 87 opal_atomic_lock(&class_lock); >>>>> (gdb) >>>>> opal_atomic_lock (lock=0xffffffff7ee89bf0 <class_lock>) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:397 >>>>> 397 while( !opal_atomic_cmpset_acq_32( &(lock->u.lock), >>>>> (gdb) >>>>> opal_atomic_cmpset_acq_32 (addr=0xffffffff7ee89bf0 <class_lock>, > oldval=0, >>>>> newval=1) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:107 >>>>> 107 rc = opal_atomic_cmpset_32(addr, oldval, newval); >>>>> (gdb) >>>>> opal_atomic_cmpset_32 (addr=0xffffffff7ee89bf0 <class_lock>, oldval=0, > newval=1) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:93 >>>>> 93 int32_t ret = newval; >>>>> (gdb) >>>>> 95 __asm__ __volatile__("casa [%1] " ASI_P ", %2, %0" >>>>> (gdb) >>>>> 98 return (ret == oldval); >>>>> (gdb) >>>>> 99 } >>>>> (gdb) >>>>> opal_atomic_cmpset_acq_32 (addr=0xffffffff7ee89bf0 <class_lock>, > oldval=0, >>>>> newval=1) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:108 >>>>> 108 opal_atomic_rmb(); >>>>> (gdb) >>>>> opal_atomic_rmb () at >>>>> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:63 >>>>> 63 MEMBAR("#LoadLoad"); >>>>> (gdb) >>>>> 64 } >>>>> (gdb) >>>>> opal_atomic_cmpset_acq_32 (addr=0xffffffff7ee89bf0 <class_lock>, > oldval=0, >>>>> newval=1) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:110 >>>>> 110 return rc; >>>>> (gdb) >>>>> 111 } >>>>> (gdb) >>>>> opal_atomic_lock (lock=0xffffffff7ee89bf0 <class_lock>) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:403 >>>>> 403 } >>>>> (gdb) >>>>> opal_class_initialize (cls=0xffffffff7f14c7d8 > <orte_state_caddy_t_class>) >>>>> at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:93 >>>>> 93 if (1 == cls->cls_initialized) { >>>>> (gdb) >>>>> 103 cls->cls_depth = 0; >>>>> (gdb) >>>>> 104 cls_construct_array_count = 0; >>>>> (gdb) >>>>> 105 cls_destruct_array_count = 0; >>>>> (gdb) >>>>> 106 for (c = cls; c; c = c->cls_parent) { >>>>> (gdb) >>>>> 107 if( NULL != c->cls_construct ) { >>>>> (gdb) >>>>> 108 cls_construct_array_count++; >>>>> (gdb) >>>>> 110 if( NULL != c->cls_destruct ) { >>>>> (gdb) >>>>> 111 cls_destruct_array_count++; >>>>> (gdb) >>>>> 113 cls->cls_depth++; >>>>> (gdb) >>>>> 106 for (c = cls; c; c = c->cls_parent) { >>>>> (gdb) >>>>> 107 if( NULL != c->cls_construct ) { >>>>> (gdb) >>>>> 110 if( NULL != c->cls_destruct ) { >>>>> (gdb) >>>>> 113 cls->cls_depth++; >>>>> (gdb) >>>>> 106 for (c = cls; c; c = c->cls_parent) { >>>>> (gdb) >>>>> 122 (void > (**)(opal_object_t*))malloc((cls_construct_array_count + >>>>> (gdb) >>>>> 123 > cls_destruct_array_count + 2) >>>>> * >>>>> (gdb) >>>>> 122 (void > (**)(opal_object_t*))malloc((cls_construct_array_count + >>>>> (gdb) >>>>> 121 cls->cls_construct_array = >>>>> (gdb) >>>>> 125 if (NULL == cls->cls_construct_array) { >>>>> (gdb) >>>>> 130 cls->cls_construct_array + cls_construct_array_count + > 1; >>>>> (gdb) >>>>> 129 cls->cls_destruct_array = >>>>> (gdb) >>>>> 136 cls_construct_array = cls->cls_construct_array + >>>>> cls_construct_array_count; >>>>> (gdb) >>>>> 137 cls_destruct_array = cls->cls_destruct_array; >>>>> (gdb) >>>>> 139 c = cls; >>>>> (gdb) >>>>> 140 *cls_construct_array = NULL; /* end marker for the > constructors */ >>>>> (gdb) >>>>> 141 for (i = 0; i < cls->cls_depth; i++) { >>>>> (gdb) >>>>> 142 if( NULL != c->cls_construct ) { >>>>> (gdb) >>>>> 143 --cls_construct_array; >>>>> (gdb) >>>>> 144 *cls_construct_array = c->cls_construct; >>>>> (gdb) >>>>> 146 if( NULL != c->cls_destruct ) { >>>>> (gdb) >>>>> 147 *cls_destruct_array = c->cls_destruct; >>>>> (gdb) >>>>> 148 cls_destruct_array++; >>>>> (gdb) >>>>> 150 c = c->cls_parent; >>>>> (gdb) >>>>> 141 for (i = 0; i < cls->cls_depth; i++) { >>>>> (gdb) >>>>> 142 if( NULL != c->cls_construct ) { >>>>> (gdb) >>>>> 146 if( NULL != c->cls_destruct ) { >>>>> (gdb) >>>>> 150 c = c->cls_parent; >>>>> (gdb) >>>>> 141 for (i = 0; i < cls->cls_depth; i++) { >>>>> (gdb) >>>>> 152 *cls_destruct_array = NULL; /* end marker for the > destructors */ >>>>> (gdb) >>>>> 154 cls->cls_initialized = 1; >>>>> (gdb) >>>>> 155 save_class(cls); >>>>> (gdb) >>>>> save_class (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>) >>>>> at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:188 >>>>> 188 if (num_classes >= max_classes) { >>>>> (gdb) >>>>> 189 expand_array(); >>>>> (gdb) >>>>> expand_array () at > ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:201 >>>>> 201 max_classes += increment; >>>>> (gdb) >>>>> 202 classes = (void**)realloc(classes, sizeof(opal_class_t*) * >>>>> max_classes); >>>>> (gdb) >>>>> 203 if (NULL == classes) { >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 208 classes[i] = NULL; >>>>> (gdb) >>>>> 207 for (i = num_classes; i < max_classes; ++i) { >>>>> (gdb) >>>>> 210 } >>>>> (gdb) >>>>> save_class (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>) >>>>> at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:192 >>>>> 192 classes[num_classes] = cls->cls_construct_array; >>>>> (gdb) >>>>> 193 ++num_classes; >>>>> (gdb) >>>>> 194 } >>>>> (gdb) >>>>> opal_class_initialize (cls=0xffffffff7f14c7d8 > <orte_state_caddy_t_class>) >>>>> at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:159 >>>>> 159 opal_atomic_unlock(&class_lock); >>>>> (gdb) >>>>> opal_atomic_unlock (lock=0xffffffff7ee89bf0 <class_lock>) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:409 >>>>> 409 opal_atomic_wmb(); >>>>> (gdb) >>>>> opal_atomic_wmb () at >>>>> ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:69 >>>>> 69 MEMBAR("#StoreStore"); >>>>> (gdb) >>>>> 70 } >>>>> (gdb) >>>>> opal_atomic_unlock (lock=0xffffffff7ee89bf0 <class_lock>) >>>>> at > ../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:410 >>>>> 410 lock->u.lock=OPAL_ATOMIC_UNLOCKED; >>>>> (gdb) >>>>> 411 } >>>>> (gdb) >>>>> opal_class_initialize (cls=0xffffffff7f14c7d8 > <orte_state_caddy_t_class>) >>>>> at ../../openmpi-dev-124-g91e9686/opal/class/opal_object.c:160 >>>>> 160 } >>>>> (gdb) >>>>> opal_obj_new (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:475 >>>>> 475 if (NULL != object) { >>>>> (gdb) >>>>> 476 object->obj_class = cls; >>>>> (gdb) >>>>> 477 object->obj_reference_count = 1; >>>>> (gdb) >>>>> 478 opal_obj_run_constructors(object); >>>>> (gdb) >>>>> opal_obj_run_constructors (object=0x1001bfcf0) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:420 >>>>> 420 assert(NULL != object->obj_class); >>>>> (gdb) >>>>> 422 cls_construct = object->obj_class->cls_construct_array; >>>>> (gdb) >>>>> 423 while( NULL != *cls_construct ) { >>>>> (gdb) >>>>> 424 (*cls_construct)(object); >>>>> (gdb) >>>>> orte_state_caddy_construct (caddy=0x1001bfcf0) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_frame.c:84 >>>>> 84 memset(&caddy->ev, 0, sizeof(opal_event_t)); >>>>> (gdb) >>>>> 85 caddy->jdata = NULL; >>>>> (gdb) >>>>> 86 } >>>>> (gdb) >>>>> opal_obj_run_constructors (object=0x1001bfcf0) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:425 >>>>> 425 cls_construct++; >>>>> (gdb) >>>>> 423 while( NULL != *cls_construct ) { >>>>> (gdb) >>>>> 427 } >>>>> (gdb) >>>>> opal_obj_new (cls=0xffffffff7f14c7d8 <orte_state_caddy_t_class>) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:480 >>>>> 480 return object; >>>>> (gdb) >>>>> 481 } >>>>> (gdb) >>>>> opal_obj_new_debug (type=0xffffffff7f14c7d8 <orte_state_caddy_t_class>, >>>>> file=0xffffffff7f034c08 >>>>> > "../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c", >>>>> line=62) at > ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:250 >>>>> 250 object->obj_magic_id = OPAL_OBJ_MAGIC_ID; >>>>> (gdb) >>>>> 251 object->cls_init_file_name = file; >>>>> (gdb) >>>>> 252 object->cls_init_lineno = line; >>>>> (gdb) >>>>> 253 return object; >>>>> (gdb) >>>>> 254 } >>>>> (gdb) >>>>> orte_state_base_activate_job_state (jdata=0x100125250, state=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:63 >>>>> 63 if (NULL != jdata) { >>>>> (gdb) >>>>> 64 caddy->jdata = jdata; >>>>> (gdb) >>>>> 65 caddy->job_state = state; >>>>> (gdb) >>>>> 66 OBJ_RETAIN(jdata); >>>>> (gdb) >>>>> opal_obj_update (inc=1, object=0x100125250) >>>>> at ../../../../openmpi-dev-124-g91e9686/opal/class/opal_object.h:497 >>>>> 497 return opal_atomic_add_32(&(object->obj_reference_count), > inc); >>>>> (gdb) >>>>> opal_atomic_add_32 (addr=0x100125260, delta=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:63 >>>>> 63 oldval = *addr; >>>>> (gdb) >>>>> 64 } while (0 == opal_atomic_cmpset_32(addr, oldval, oldval + > delta)); >>>>> (gdb) >>>>> opal_atomic_cmpset_32 (addr=0x100125260, oldval=1, newval=2) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/opal/include/opal/sys/sparcv9/atomic.h:93 >>>>> 93 int32_t ret = newval; >>>>> (gdb) >>>>> 95 __asm__ __volatile__("casa [%1] " ASI_P ", %2, %0" >>>>> (gdb) >>>>> 98 return (ret == oldval); >>>>> (gdb) >>>>> 99 } >>>>> (gdb) >>>>> opal_atomic_add_32 (addr=0x100125260, delta=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/opal/include/opal/sys/atomic_impl.h:65 >>>>> 65 return (oldval + delta); >>>>> (gdb) >>>>> 66 } >>>>> (gdb) >>>>> orte_state_base_activate_job_state (jdata=0x100125250, state=1) >>>>> at >>>>> > ../../../../openmpi-dev-124-g91e9686/orte/mca/state/base/state_base_fns.c:66 >>>>> 66 OBJ_RETAIN(jdata); >>>>> (gdb) >>>>> 68 opal_event_set(orte_event_base, &caddy->ev, -1, >>>>> OPAL_EV_WRITE, s->cbfunc, caddy); >>>>> (gdb) >>>>> 69 opal_event_set_priority(&caddy->ev, s->priority); >>>>> (gdb) >>>>> 70 opal_event_active(&caddy->ev, OPAL_EV_WRITE, 1); >>>>> (gdb) >>>>> 71 return; >>>>> (gdb) >>>>> 105 } >>>>> (gdb) >>>>> rsh_launch (jdata=0x100125250) >>>>> at >>>>> > ../../../../../openmpi-dev-124-g91e9686/orte/mca/plm/rsh/plm_rsh_module.c:883 >>>>> 883 return ORTE_SUCCESS; >>>>> (gdb) >>>>> 884 } >>>>> (gdb) >>>>> orterun (argc=5, argv=0xffffffff7fffe0d8) >>>>> at > ../../../../openmpi-dev-124-g91e9686/orte/tools/orterun/orterun.c:1084 >>>>> 1084 while (orte_event_base_active) { >>>>> (gdb) >>>>> 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE); >>>>> (gdb) >>>>> 1084 while (orte_event_base_active) { >>>>> (gdb) >>>>> 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE); >>>>> (gdb) >>>>> 1084 while (orte_event_base_active) { >>>>> (gdb) >>>>> 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE); >>>>> (gdb) >>>>> # >>>>> # A fatal error has been detected by the Java Runtime Environment: >>>>> # >>>>> # SIGSEGV (0xb) at pc=0xffffffff7ea3c7f0, pid=13080, tid=2 >>>>> # >>>>> # JRE version: Java(TM) SE Runtime Environment (8.0-b132) (build > 1.8.0-b132) >>>>> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode > solaris-sparc >>>>> compressed oops) >>>>> # Problematic frame: >>>>> # 1084 while (orte_event_base_active) { >>>>> (gdb) >>>>> 1085 opal_event_loop(orte_event_base, OPAL_EVLOOP_ONCE); >>>>> (gdb) >>>>> C [libc.so.1+0x3c7f0] strlen+0x50 >>>>> # >>>>> # Failed to write core dump. Core dumps have been disabled. To enable > core >>>>> dumping, try "ulimit -c unlimited" before starting Java again >>>>> # >>>>> # An error report file with more information is saved as: >>>>> # > /home/fd1026/work/skripte/master/parallel/prog/mpi/java/hs_err_pid13080.log >>>>> # >>>>> # If you would like to submit a bug report, please visit: >>>>> # http://bugreport.sun.com/bugreport/crash.jsp >>>>> # The crash happened outside the Java Virtual Machine in native code. >>>>> # See problematic frame for where to report the bug. >>>>> # >>>>> > -------------------------------------------------------------------------- >>>>> mpiexec noticed that process rank 0 with PID 0 on node tyr exited on > signal 6 >>>>> (Abort). >>>>> > -------------------------------------------------------------------------- >>>>> 1084 while (orte_event_base_active) { >>>>> (gdb) >>>>> 1089 orte_odls.kill_local_procs(NULL); >>>>> (gdb) >>>>> >>>>> >>>>> Thank you very much for any help in advance. >>>>> >>>>> Kind regards >>>>> >>>>> Siegmar >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: > http://www.open-mpi.org/community/lists/users/2014/10/25559.php > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/10/25562.php
diff --git a/ompi/mpi/java/c/mpi_MPI.c b/ompi/mpi/java/c/mpi_MPI.c index 7c3a3ba..219da6e 100644 --- a/ompi/mpi/java/c/mpi_MPI.c +++ b/ompi/mpi/java/c/mpi_MPI.c @@ -62,6 +62,7 @@ #include <sys/stat.h> #endif #include <dlfcn.h> +#include <poll.h> #include "opal/util/output.h" #include "opal/datatype/opal_convertor.h" @@ -121,6 +122,11 @@ OBJ_CLASS_INSTANCE(ompi_java_buffer_t, */ jint JNI_OnLoad(JavaVM *vm, void *reserved) { + char *env = getenv("OMPI_ATTACH"); + if (NULL != env && 0 < atoi(env)) { + volatile int _dbg = 1; + while (_dbg) poll(NULL, 0, 1); + } libmpi = dlopen("libmpi." OPAL_DYN_LIB_SUFFIX, RTLD_NOW | RTLD_GLOBAL); if(libmpi == NULL)