Okay, I finally had time to parse this and fix it. Thanks!
On May 16, 2011, at 1:02 PM, Peter Thompson wrote:
> Hmmm? We're not removing the putenv() calls. Just adding a strdup()
> beforehand, and then calling putenv() with the string duplicated from env[j].
> Of course, if the strdup fails, then we bail out.
> As for why it's suddenly a problem, I'm not quite as certain. The problem
> we do show is a double free, so someone has already freed that memory used by
> putenv(), and I do know that while that used to be just flagged as an event
> before, now we seem to be unable to continue past it. Not sure if that is
> our change or a library/system change.
> PeterT
>
>
> Ralph Castain wrote:
>> On May 16, 2011, at 12:45 PM, Peter Thompson wrote:
>>
>>
>>> Hi Ralph,
>>>
>>> We've had a number of user complaints about this. Since it seems on the
>>> face of it that it is a debugger issue, it may have not made it's way back
>>> here. Is your objection that the patch basically aborts if it gets a bad
>>> value? I could understand that being a concern. Of course, it aborts on
>>> TotalView now if we attempt to move forward without this patch.
>>>
>>>
>>
>> No - my concern is that you appear to be removing the "putenv" calls. OMPI
>> places some values into the local environment so the user can control
>> behavior. Removing those causes problems.
>>
>> What I need to know is why, after it has worked with TV for years, these
>> putenv's are suddenly a problem. Is the problem occurring during shutdown?
>> Or is this something that causes TV to break?
>>
>>
>>
>>> I've passed your comment back to the engineer, with a suspicion about the
>>> concerns about the abort, but if you have other objections, let me know.
>>>
>>> Cheers,
>>> PeterT
>>>
>>>
>>> Ralph Castain wrote:
>>>
>>>> That would be a problem, I fear. We need to push those envars into the
>>>> environment.
>>>>
>>>> Is there some particular problem causing what you see? We have no other
>>>> reports of this issue, and orterun has had that code forever.
>>>>
>>>>
>>>>
>>>> Sent from my iPad
>>>>
>>>> On May 11, 2011, at 2:05 PM, Peter Thompson <peter.thomp...@roguewave.com>
>>>> wrote:
>>>>
>>>>
>>>>> We've gotten a few reports of problems with memory debugging when using
>>>>> OpenMPI under TotalView. Usually, TotalView will attach tot he processes
>>>>> started after an MPI_Init. However in the case where memory debugging is
>>>>> enabled, things seemed to run away or fail. My analysis showed that we
>>>>> had a number of core files left over from the attempt, and all were
>>>>> mpirun (or orterun) cores. It seemed to be a regression on our part,
>>>>> since testing seemed to indicate this worked okay before TotalView
>>>>> 8.9.0-0, so I filed an internal bug and passed it to engineering. After
>>>>> giving our engineer a brief tutorial on how to build a debug version of
>>>>> OpenMPI, he found what appears to be a problem in the code for orterun.c.
>>>>> He's made a slight change that fixes the issue in 1.4.2, 1.4.3,
>>>>> 1.4.4rc2 and 1.5.3, those being the versions he's tested with so far.
>>>>> He doesn't subscribe to this list that I know of, so I offered to pass
>>>>> this by the group. Of course, I'm not sure if this is exactly the right
>>>>> place to submit patches, but I'm sure you'd tell me where to put it if
>>>>> I'm in the wrong here. It's a short patch, so I'll cut and paste it,
>>>>> and attach as well, since cut and paste can do weird things to formatting.
>>>>>
>>>>> Credit goes to Ariel Burton for this patch. Of course he used TotalVIew
>>>>> to find this ;-) It shows up if you do 'mpirun -tv -np 4 ./foo' or
>>>>> 'totalview mpirun -a -np 4 ./foo'
>>>>>
>>>>> Cheers,
>>>>> PeterT
>>>>>
>>>>>
>>>>> more ~/patches/anbs-patch
>>>>> *** orte/tools/orterun/orterun.c 2010-04-13 13:30:34.000000000
>>>>> -0400
>>>>> ---
>>>>> /home/anb/packages/openmpi-1.4.2/linux-x8664-iwashi/installation/bin/../../.
>>>>> ./src/openmpi-1.4.2/orte/tools/orterun/orterun.c 2011-05-09
>>>>> 20:28:16.5881
>>>>> 83000 -0400
>>>>> ***************
>>>>> *** 1578,1588 ****
>>>>> }
>>>>> if (NULL != env) {
>>>>> size1 = opal_argv_count(env);
>>>>> for (j = 0; j < size1; ++j) {
>>>>> ! putenv(env[j]);
>>>>> }
>>>>> }
>>>>> /* All done */
>>>>> --- 1578,1600 ----
>>>>> }
>>>>> if (NULL != env) {
>>>>> size1 = opal_argv_count(env);
>>>>> for (j = 0; j < size1; ++j) {
>>>>> ! /* Use-after-Free error possible here. putenv does not copy
>>>>> ! the string passed to it, and instead stores only the
>>>>> pointer.
>>>>> ! env[j] may be freed later, in which case the pointer
>>>>> ! in environ will now be left dangling into a deallocated
>>>>> ! region.
>>>>> ! So we make a copy of the variable.
>>>>> ! */
>>>>> ! char *s = strdup(env[j]);
>>>>> !
>>>>> ! if (NULL == s) {
>>>>> ! return OPAL_ERR_OUT_OF_RESOURCE;
>>>>> ! }
>>>>> ! putenv(s);
>>>>> }
>>>>> }
>>>>> /* All done */
>>>>>
>>>>> *** orte/tools/orterun/orterun.c 2010-04-13 13:30:34.000000000 -0400
>>>>> ---
>>>>> /home/anb/packages/openmpi-1.4.2/linux-x8664-iwashi/installation/bin/../../../src/openmpi-1.4.2/orte/tools/orterun/orterun.c
>>>>> 2011-05-09 20:28:16.588183000 -0400
>>>>> ***************
>>>>> *** 1578,1588 ****
>>>>> }
>>>>>
>>>>> if (NULL != env) {
>>>>> size1 = opal_argv_count(env);
>>>>> for (j = 0; j < size1; ++j) {
>>>>> ! putenv(env[j]);
>>>>> }
>>>>> }
>>>>>
>>>>> /* All done */
>>>>>
>>>>> --- 1578,1600 ----
>>>>> }
>>>>>
>>>>> if (NULL != env) {
>>>>> size1 = opal_argv_count(env);
>>>>> for (j = 0; j < size1; ++j) {
>>>>> ! /* Use-after-Free error possible here. putenv does not copy
>>>>> ! the string passed to it, and instead stores only the
>>>>> pointer.
>>>>> ! env[j] may be freed later, in which case the pointer
>>>>> ! in environ will now be left dangling into a deallocated
>>>>> ! region.
>>>>> ! So we make a copy of the variable.
>>>>> ! */
>>>>> ! char *s = strdup(env[j]);
>>>>> ! ! if (NULL == s) {
>>>>> ! return OPAL_ERR_OUT_OF_RESOURCE;
>>>>> ! }
>>>>> ! putenv(s);
>>>>> }
>>>>> }
>>>>>
>>>>> /* All done */
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>
>>
>
