Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
after removing my debug prints - the correct line is 448 On Mon, Dec 12, 2011 at 7:46 PM, Mike Dubman wrote: > > > Hi guys, > > The latest v1.5 from trunk, compiled in debug mode yields following error > with openib. > The quick blame leads to the following commit: > > r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 lines > Fixes #2930: CMR: memcpy() w/overlapping addresses in > btl_openib_connect_oob.c > Patch attached to the ticket. > > > > JOB MAP > > Data for node: boo12 Num procs: 8 > Process OMPI jobid: [35908,1] Process rank: 0 > Process OMPI jobid: [35908,1] Process rank: 2 > Process OMPI jobid: [35908,1] Process rank: 4 > Process OMPI jobid: [35908,1] Process rank: 6 > Process OMPI jobid: [35908,1] Process rank: 8 > Process OMPI jobid: [35908,1] Process rank: 10 > Process OMPI jobid: [35908,1] Process rank: 12 > Process OMPI jobid: [35908,1] Process rank: 14 > > Data for node: boo11 Num procs: 8 > Process OMPI jobid: [35908,1] Process rank: 1 > Process OMPI jobid: [35908,1] Process rank: 3 > Process OMPI jobid: [35908,1] Process rank: 5 > Process OMPI jobid: [35908,1] Process rank: 7 > Process OMPI jobid: [35908,1] Process rank: 9 > Process OMPI jobid: [35908,1] Process rank: 11 > Process OMPI jobid: [35908,1] Process rank: 13 > Process OMPI jobid: [35908,1] Process rank: 15 > = > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > FYI > M >
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
Do we have the same issue in the trunk? george. On Dec 12, 2011, at 12:49 , Mike Dubman wrote: > after removing my debug prints - the correct line is 448 > > On Mon, Dec 12, 2011 at 7:46 PM, Mike Dubman wrote: > > > Hi guys, > > The latest v1.5 from trunk, compiled in debug mode yields following error > with openib. > The quick blame leads to the following commit: > > r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 lines > Fixes #2930: CMR: memcpy() w/overlapping addresses in btl_openib_connect_oob.c > Patch attached to the ticket. > > > > JOB MAP > Data for node: boo12 Num procs: 8 > Process OMPI jobid: [35908,1] Process rank: 0 > Process OMPI jobid: [35908,1] Process rank: 2 > Process OMPI jobid: [35908,1] Process rank: 4 > Process OMPI jobid: [35908,1] Process rank: 6 > Process OMPI jobid: [35908,1] Process rank: 8 > Process OMPI jobid: [35908,1] Process rank: 10 > Process OMPI jobid: [35908,1] Process rank: 12 > Process OMPI jobid: [35908,1] Process rank: 14 > > Data for node: boo11 Num procs: 8 > Process OMPI jobid: [35908,1] Process rank: 1 > Process OMPI jobid: [35908,1] Process rank: 3 > Process OMPI jobid: [35908,1] Process rank: 5 > Process OMPI jobid: [35908,1] Process rank: 7 > Process OMPI jobid: [35908,1] Process rank: 9 > Process OMPI jobid: [35908,1] Process rank: 11 > Process OMPI jobid: [35908,1] Process rank: 13 > Process OMPI jobid: [35908,1] Process rank: 15 > > = > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > malloc debug: Invalid free (btl_openib_endpoint.c, 452) > FYI > M > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
nope On Mon, Dec 12, 2011 at 10:40 PM, George Bosilca wrote: > Do we have the same issue in the trunk? > > george. > > On Dec 12, 2011, at 12:49 , Mike Dubman wrote: > > after removing my debug prints - the correct line is 448 > > On Mon, Dec 12, 2011 at 7:46 PM, Mike Dubman wrote: > >> >> >> Hi guys, >> >> The latest v1.5 from trunk, compiled in debug mode yields following error >> with openib. >> The quick blame leads to the following commit: >> >> r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 lines >> Fixes #2930: CMR: memcpy() w/overlapping addresses in >> btl_openib_connect_oob.c >> Patch attached to the ticket. >> >> >> >> JOB MAP >> >> Data for node: boo12 Num procs: 8 >> Process OMPI jobid: [35908,1] Process rank: 0 >> Process OMPI jobid: [35908,1] Process rank: 2 >> Process OMPI jobid: [35908,1] Process rank: 4 >> Process OMPI jobid: [35908,1] Process rank: 6 >> Process OMPI jobid: [35908,1] Process rank: 8 >> Process OMPI jobid: [35908,1] Process rank: 10 >> Process OMPI jobid: [35908,1] Process rank: 12 >> Process OMPI jobid: [35908,1] Process rank: 14 >> >> Data for node: boo11 Num procs: 8 >> Process OMPI jobid: [35908,1] Process rank: 1 >> Process OMPI jobid: [35908,1] Process rank: 3 >> Process OMPI jobid: [35908,1] Process rank: 5 >> Process OMPI jobid: [35908,1] Process rank: 7 >> Process OMPI jobid: [35908,1] Process rank: 9 >> Process OMPI jobid: [35908,1] Process rank: 11 >> Process OMPI jobid: [35908,1] Process rank: 13 >> Process OMPI jobid: [35908,1] Process rank: 15 >> = >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> FYI >> M >> > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
We are investigating. A fix will be hopefully provided soon. Thanks for the report, george. On Dec 13, 2011, at 00:25 , Mike Dubman wrote: > nope > > On Mon, Dec 12, 2011 at 10:40 PM, George Bosilca wrote: > Do we have the same issue in the trunk? > > george. > > On Dec 12, 2011, at 12:49 , Mike Dubman wrote: > >> after removing my debug prints - the correct line is 448 >> >> On Mon, Dec 12, 2011 at 7:46 PM, Mike Dubman wrote: >> >> >> Hi guys, >> >> The latest v1.5 from trunk, compiled in debug mode yields following error >> with openib. >> The quick blame leads to the following commit: >> >> r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 lines >> Fixes #2930: CMR: memcpy() w/overlapping addresses in >> btl_openib_connect_oob.c >> Patch attached to the ticket. >> >> >> >> JOB MAP >> Data for node: boo12 Num procs: 8 >> Process OMPI jobid: [35908,1] Process rank: 0 >> Process OMPI jobid: [35908,1] Process rank: 2 >> Process OMPI jobid: [35908,1] Process rank: 4 >> Process OMPI jobid: [35908,1] Process rank: 6 >> Process OMPI jobid: [35908,1] Process rank: 8 >> Process OMPI jobid: [35908,1] Process rank: 10 >> Process OMPI jobid: [35908,1] Process rank: 12 >> Process OMPI jobid: [35908,1] Process rank: 14 >> >> Data for node: boo11 Num procs: 8 >> Process OMPI jobid: [35908,1] Process rank: 1 >> Process OMPI jobid: [35908,1] Process rank: 3 >> Process OMPI jobid: [35908,1] Process rank: 5 >> Process OMPI jobid: [35908,1] Process rank: 7 >> Process OMPI jobid: [35908,1] Process rank: 9 >> Process OMPI jobid: [35908,1] Process rank: 11 >> Process OMPI jobid: [35908,1] Process rank: 13 >> Process OMPI jobid: [35908,1] Process rank: 15 >> >> = >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >> FYI >> M >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
btw, it appears, that problem happens when CFLAGS="-g -O0" during configure phase. another clone compiled w/o this option did not cry. On Tue, Dec 13, 2011 at 6:10 PM, George Bosilca wrote: > We are investigating. A fix will be hopefully provided soon. > > Thanks for the report, > george. > > On Dec 13, 2011, at 00:25 , Mike Dubman wrote: > > nope > > On Mon, Dec 12, 2011 at 10:40 PM, George Bosilca wrote: > >> Do we have the same issue in the trunk? >> >> george. >> >> On Dec 12, 2011, at 12:49 , Mike Dubman wrote: >> >> after removing my debug prints - the correct line is 448 >> >> On Mon, Dec 12, 2011 at 7:46 PM, Mike Dubman wrote: >> >>> >>> >>> Hi guys, >>> >>> The latest v1.5 from trunk, compiled in debug mode yields following >>> error with openib. >>> The quick blame leads to the following commit: >>> >>> r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 lines >>> Fixes #2930: CMR: memcpy() w/overlapping addresses in >>> btl_openib_connect_oob.c >>> Patch attached to the ticket. >>> >>> >>> >>> JOB MAP >>> >>> Data for node: boo12 Num procs: 8 >>> Process OMPI jobid: [35908,1] Process rank: 0 >>> Process OMPI jobid: [35908,1] Process rank: 2 >>> Process OMPI jobid: [35908,1] Process rank: 4 >>> Process OMPI jobid: [35908,1] Process rank: 6 >>> Process OMPI jobid: [35908,1] Process rank: 8 >>> Process OMPI jobid: [35908,1] Process rank: 10 >>> Process OMPI jobid: [35908,1] Process rank: 12 >>> Process OMPI jobid: [35908,1] Process rank: 14 >>> >>> Data for node: boo11 Num procs: 8 >>> Process OMPI jobid: [35908,1] Process rank: 1 >>> Process OMPI jobid: [35908,1] Process rank: 3 >>> Process OMPI jobid: [35908,1] Process rank: 5 >>> Process OMPI jobid: [35908,1] Process rank: 7 >>> Process OMPI jobid: [35908,1] Process rank: 9 >>> Process OMPI jobid: [35908,1] Process rank: 11 >>> Process OMPI jobid: [35908,1] Process rank: 13 >>> Process OMPI jobid: [35908,1] Process rank: 15 >>> = >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> FYI >>> M >>> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel >
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
We decided on the call to just roll this back for 1.5.5, and fully investigate/fix for 1.6. On Dec 13, 2011, at 3:10 PM, Mike Dubman wrote: > btw, it appears, that problem happens when CFLAGS="-g -O0" during configure > phase. > another clone compiled w/o this option did not cry. > > On Tue, Dec 13, 2011 at 6:10 PM, George Bosilca wrote: > We are investigating. A fix will be hopefully provided soon. > > Thanks for the report, > george. > > On Dec 13, 2011, at 00:25 , Mike Dubman wrote: > >> nope >> >> On Mon, Dec 12, 2011 at 10:40 PM, George Bosilca >> wrote: >> Do we have the same issue in the trunk? >> >> george. >> >> On Dec 12, 2011, at 12:49 , Mike Dubman wrote: >> >>> after removing my debug prints - the correct line is 448 >>> >>> On Mon, Dec 12, 2011 at 7:46 PM, Mike Dubman wrote: >>> >>> >>> Hi guys, >>> >>> The latest v1.5 from trunk, compiled in debug mode yields following error >>> with openib. >>> The quick blame leads to the following commit: >>> >>> r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 lines >>> Fixes #2930: CMR: memcpy() w/overlapping addresses in >>> btl_openib_connect_oob.c >>> Patch attached to the ticket. >>> >>> >>> >>> JOB MAP >>> Data for node: boo12 Num procs: 8 >>> Process OMPI jobid: [35908,1] Process rank: 0 >>> Process OMPI jobid: [35908,1] Process rank: 2 >>> Process OMPI jobid: [35908,1] Process rank: 4 >>> Process OMPI jobid: [35908,1] Process rank: 6 >>> Process OMPI jobid: [35908,1] Process rank: 8 >>> Process OMPI jobid: [35908,1] Process rank: 10 >>> Process OMPI jobid: [35908,1] Process rank: 12 >>> Process OMPI jobid: [35908,1] Process rank: 14 >>> >>> Data for node: boo11 Num procs: 8 >>> Process OMPI jobid: [35908,1] Process rank: 1 >>> Process OMPI jobid: [35908,1] Process rank: 3 >>> Process OMPI jobid: [35908,1] Process rank: 5 >>> Process OMPI jobid: [35908,1] Process rank: 7 >>> Process OMPI jobid: [35908,1] Process rank: 9 >>> Process OMPI jobid: [35908,1] Process rank: 11 >>> Process OMPI jobid: [35908,1] Process rank: 13 >>> Process OMPI jobid: [35908,1] Process rank: 15 >>> >>> = >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> malloc debug: Invalid free (btl_openib_endpoint.c, 452) >>> FYI >>> M >>> >>> ___ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
Hi Mike, On Mon, 12 Dec 2011 19:46:08 +0200 Mike Dubman wrote: > Hi guys, > > The latest v1.5 from trunk, compiled in debug mode yields following > error with openib. > The quick blame leads to the following commit: > > r25616 | bosilca | 2011-12-10 00:18:16 +0200 (Sat, 10 Dec 2011) | 4 > lines Fixes #2930: CMR: memcpy() w/overlapping addresses in > btl_openib_connect_oob.c > Patch attached to the ticket. > Sorry, late to the discussion. This is a spurious warning caused by passing the NULL pointer to the opal free function which is actually ok. It was fixed by #2884 - this is why you don't see the problem in trunk. I'll put in CMRs for 1.4/1.5 Regards, Chris -- cy...@ozlabs.org
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
On Dec 13, 2011, at 7:59 PM, Christopher Yeoh wrote: > Sorry, late to the discussion. This is a spurious warning caused by > passing the NULL pointer to the opal free function which is actually ok. > It was fixed by #2884 - this is why you don't see the problem in trunk. I have a dim recollection that we talked about this before, but I'm going to ask again, anyway... https://svn.open-mpi.org/trac/ompi/changeset/25430 Why are we removing this debug message? It seems like a useful thing. It might well be legal to call free(NULL), but it seems like a bad idea to call free if we know the value will be NULL. Did we talk about this before, and I was in the minority for thinking removing it was a bad idea? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] Invalid free (btl_openib_endpoint.c, 448) in v1.5
On Tue, 13 Dec 2011 20:27:00 -0500 Jeff Squyres wrote: > On Dec 13, 2011, at 7:59 PM, Christopher Yeoh wrote: > > > Sorry, late to the discussion. This is a spurious warning caused by > > passing the NULL pointer to the opal free function which is > > actually ok. It was fixed by #2884 - this is why you don't see the > > problem in trunk. > > I have a dim recollection that we talked about this before, but I'm > going to ask again, anyway... > > https://svn.open-mpi.org/trac/ompi/changeset/25430 > > Why are we removing this debug message? It seems like a useful > thing. It might well be legal to call free(NULL), but it seems like > a bad idea to call free if we know the value will be NULL. Its more that the pointer value might be quite validly NULL. For example in this case that is causing the spurious warning a member of a struct is set to NULL initially and may or may not be allocated memory during the lifetime of the struct, and if it is may be cleaned up earlier on by another function and the value set back to NULL. So during the final cleanup of structs that behave this way, we can have this: if (NULL != foo->pointer) free(foo->pointer); In the case where we have a non NULL pointer we end up doing two checks since the free function already does a NULL pointer check at the start before proceeding. Or we can just do: free(foo->pointer) If its in a performance critical area and we think that the NULL pointer case is going to be very common and not calling free would make a difference we can always add a check before calling free. > Did we talk about this before, and I was in the minority for thinking > removing it was a bad idea? I don't think we talked much about this before (maybe you are thinking of the discussion around use of an assert?). There wasn't much discussion on the ticket https://svn.open-mpi.org/trac/ompi/ticket/2884 -- cy...@ozlabs.org
