Hi, > Hi Rafael -- > > Thanks for your analysis and description.
Thanks to you and all that have help with this problem, I have been wrestling with it a long time, and at the end left it for impossible at the beginning of the year. I have just checked that my code works now fine with ibv, it feels good to know it was not a bug in my code :-) > I don't have any problem with changing the default segment for 'ibv' and > agree that it sounds like the right thing to do if it works around this > issue. It seems like 'fast' would be preferable to 'large' from a > performance perspective (based on a quick glance at the docs -- I'm mostly > unfamiliar with 'large'). But fast doesn't work with ibv, it shows this error: *** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when registering the segment So I think we should stay with large on ibv, I have not benchmarked its speed impact, but correct bulks comm should be enought to offset some impact on speed. > In trying to use the 'fast' segment, do you know whether the memory > allocator used changed to either tcmalloc or dlmalloc (either > automatically or manually)? That is probably necessary for correctness, > but should've (hopefully) happened automatically. printchplenv can be used > to verify. Yes, it automatically uses dlmalloc as explained in doc/release/README.chplenv, I have checked it. > Note that I believe you should only need to set CHPL_GASNET_SEGMENT. The > CHPL_MAKE_ variables should follow automatically (and are not intended to > be set by the end-user). Yes, you are right, it seems that using only : export CHPL_GASNET_SEGMENT=large works fine, but I worked by reverse-engeneering :-) Thanks to all, Rafael > Thanks again, > -Brad > > On Thu, 30 Jan 2014, Rafael Larrosa Jiménez wrote: > > El Jueves, 30 de enero de 2014 12:21:11 Akihiro Hayashi escribió: > >> Hi, > >> > >> Thanks for your helping me to understand the problem. > >> I think now I understand the problem correctly. > >> > >> Rafael Asenjo and Rafael Larrosa. > >> I'm pretty sure my assignment have involved a contiguous block and > >> useBulkTransfer is active and useBulkTransferStride is not active in my > >> chapel compiler. I just wanted to make both sure my code > >> (useBulkTransfer) > >> and Rafaerl Larrosa's code (useBulkTransferStride) can fail due to the > >> ibv-conduit bug (https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495). > > > > First, tell that I'm not the developer, but a client of that code. > > > > Second, I have been writting this email for several hours, as I changed my > > vision on the problem, have been reading the ib verbs manuals, ib code, > > etc. > > > > At the end I have found the solution, by default chapel puts all memory as > > memory accesible by RDMA, which seems to be the problem, as it can be > > "unpinned, if instead only "large" portions are used for RDMA then it > > works > > fine, as explained in one of the messages: > > > > --- > > In a GASNET_SEGMENT_FAST or _LARGE configuration the segment is obtained > > at > > startup via mmap() and is never unmapped. So, sending from inside the > > GASNet segment in these two cases will ensure this bug cannot occur. > > --- > > > > So if you define : > > > > export CHPL_MAKE_GASNET_SEGMENT=large > > export CHPL_MAKE_COMM_SEGMENT=large > > export CHPL_GASNET_SEGMENT=large > > > > Then do a make clobber followed by a make and recompile your program, it > > should work fine. > > > > BTW, when using portals, gemini or aries, the fast segment is used, as > > explained in : > > > > doc/release/README.multilocale > > --- > > 3) Advanced GASNet users can set CHPL_GASNET_SEGMENT to choose a > > > > memory segment to use with GASNet. Current defaults are: > > When CHPL_COMM_SUBSTRATE is... Chapel will choose... > > > > portals fast > > gemini fast > > (other) everything > > > > --- > > > > But using fast with ibv gives this error: > > *** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when > > registering the segment > > > > But that is not a problem as large segments are the solution, hope they > > don't break other things :-) > > > > I have tested it with Akihiro code, and it seems to work fine now. > > > > Perphaps is a good idea to change the default to use large segments with > > ibv. > > > > Greets, > > > > Rafael > > > > EMAIL: [email protected] Edificio de Bioinnovación > > TELEF: + 34951952788 C/ Severo Ochoa 34 > > FAX : +34951952792 Parque Tecnológico de Andalucía > > > > 29590 Málaga (SPAIN) -- Rafael Larrosa Jiménez Centro de Supercomputación y Bioinformática - http://www.scbi.uma.es Universidad de Málaga EMAIL: [email protected] Edificio de Bioinnovación TELEF: + 34951952788 C/ Severo Ochoa 34 FAX : +34951952792 Parque Tecnológico de Andalucía 29590 Málaga (SPAIN) ------------------------------------------------------------------------------ WatchGuard Dimension instantly turns raw network data into actionable security intelligence. It gives you real-time visual feedback on key security issues and trends. Skip the complicated setup - simply import a virtual appliance and go from zero to informed in seconds. http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk _______________________________________________ Chapel-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/chapel-developers
