Hi, > Hi Rafael -- > > Here's another tip from the GASNet team. Would you be able to give this a > try and see if it helps with the ibv + fast? > > ---- > > The Bad Address sounds like an interaction between IBV and POSIX Shared > Memory: > https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=2929 > > The work-around is to configure GASNet with "--enable-pshm-sysv > --disable-pshm-posix". > > ----
Yes, it works fine with those changes, sorry for the delay. Greets, Rafael > Thanks, > -Brad > > On Thu, 30 Jan 2014, Brad Chamberlain wrote: > > Hi Rafael -- > > > > Thanks for your analysis and description. > > > > I don't have any problem with changing the default segment for 'ibv' and > > agree that it sounds like the right thing to do if it works around this > > issue. It seems like 'fast' would be preferable to 'large' from a > > performance perspective (based on a quick glance at the docs -- I'm mostly > > unfamiliar with 'large'). > > > > In trying to use the 'fast' segment, do you know whether the memory > > allocator used changed to either tcmalloc or dlmalloc (either > > automatically or manually)? That is probably necessary for correctness, > > but should've (hopefully) happened automatically. printchplenv can be > > used to verify. > > > > Note that I believe you should only need to set CHPL_GASNET_SEGMENT. The > > CHPL_MAKE_ variables should follow automatically (and are not intended to > > be set by the end-user). > > > > Thanks again, > > -Brad > > > > On Thu, 30 Jan 2014, Rafael Larrosa Jiménez wrote: > >> El Jueves, 30 de enero de 2014 12:21:11 Akihiro Hayashi escribió: > >>> Hi, > >>> > >>> Thanks for your helping me to understand the problem. > >>> I think now I understand the problem correctly. > >>> > >>> Rafael Asenjo and Rafael Larrosa. > >>> I'm pretty sure my assignment have involved a contiguous block and > >>> useBulkTransfer is active and useBulkTransferStride is not active in my > >>> chapel compiler. I just wanted to make both sure my code > >>> (useBulkTransfer) > >>> and Rafaerl Larrosa's code (useBulkTransferStride) can fail due to the > >>> ibv-conduit bug (https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495). > >> > >> First, tell that I'm not the developer, but a client of that code. > >> > >> Second, I have been writting this email for several hours, as I changed > >> my > >> vision on the problem, have been reading the ib verbs manuals, ib code, > >> etc. > >> > >> At the end I have found the solution, by default chapel puts all memory > >> as > >> memory accesible by RDMA, which seems to be the problem, as it can be > >> "unpinned, if instead only "large" portions are used for RDMA then it > >> works > >> fine, as explained in one of the messages: > >> > >> --- > >> In a GASNET_SEGMENT_FAST or _LARGE configuration the segment is obtained > >> at > >> startup via mmap() and is never unmapped. So, sending from inside the > >> GASNet > >> segment in these two cases will ensure this bug cannot occur. > >> --- > >> > >> So if you define : > >> > >> export CHPL_MAKE_GASNET_SEGMENT=large > >> export CHPL_MAKE_COMM_SEGMENT=large > >> export CHPL_GASNET_SEGMENT=large > >> > >> Then do a make clobber followed by a make and recompile your program, it > >> should work fine. > >> > >> BTW, when using portals, gemini or aries, the fast segment is used, as > >> explained in : > >> > >> doc/release/README.multilocale > >> --- > >> 3) Advanced GASNet users can set CHPL_GASNET_SEGMENT to choose a > >> > >> memory segment to use with GASNet. Current defaults are: > >> When CHPL_COMM_SUBSTRATE is... Chapel will choose... > >> > >> portals fast > >> gemini fast > >> (other) everything > >> > >> --- > >> > >> But using fast with ibv gives this error: > >> *** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when > >> registering > >> the segment > >> > >> But that is not a problem as large segments are the solution, hope they > >> don't > >> break other things :-) > >> > >> I have tested it with Akihiro code, and it seems to work fine now. > >> > >> Perphaps is a good idea to change the default to use large segments with > >> ibv. > >> > >> Greets, > >> > >> Rafael -- Rafael Larrosa Jiménez Centro de Supercomputación y Bioinformática - http://www.scbi.uma.es Universidad de Málaga EMAIL: [email protected] Edificio de Bioinnovación TELEF: + 34951952788 C/ Severo Ochoa 34 FAX : +34951952792 Parque Tecnológico de Andalucía 29590 Málaga (SPAIN) ------------------------------------------------------------------------------ Managing the Performance of Cloud-Based Applications Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. Read the Whitepaper. http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk _______________________________________________ Chapel-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-developers
