Hi,
> Great, thanks for checking!
>
> Based on that, my thought is that for ibv we should use segment fast
> (rationale: it's faster; it's more similar to what we do elsewhere and we
> test it more) and use the configure flags below to make it work. Rafael
> (or anyone), do you see any reasons not to do this?
One of the reasons why I took some time to answer was that I wanted to
make some benchmarks before answering, but I couldn't.
Now I have made some benchmarks and have found that there are problems when
using fast and a lot of memory, the exact error is:
@ 0> snd status=5 opcode=0 dst_node=1 dst_qp=0
@ 0> - rcv CQ contains impossibly large WCE count with status 5
*** FATAL ERROR: aborting on reap of failed send
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment
to generate a backtrace.
@ 0> snd status=10 opcode=2 dst_node=1 dst_qp=0
@ 0> - rcv CQ contains impossibly large WCE count with status 10
*** FATAL ERROR: aborting on reap of failed send
*** Caught a fatal signal: SIGABRT(6) on node 0/2
*** Caught a fatal signal: SIGABRT(6) on node 0/2
@ 1> rcv comp->status=5
@ 1> - snd CQ contains impossibly large WCE count with status 5
*** FATAL ERROR: aborting on reap of failed recv
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 in the environment
to generate a backtrace.
*** Caught a fatal signal: SIGABRT(6) on node 1/2
That is using only two nodes with 64 GB of RAM each and an array of 16 GBytes
of RAM (8 GB in each node), with smaller problems it works fine, but the speed
is almost the same using segments large or fast (less than 1% in my tests),
althought almost all the time was due to overload, not in movements of data.
When using a bigger size of data speed was much higher, but only large could
be used, as fast fails.
> If not, Rafael, would this be a fairly simple patch for you to write,
> test, and submit? (we don't have easy access to an ibv cluster, which is
> part of the reason we haven't run into this ourselves). I think it should
> be as simple as adding a case to util/chplenv/commSegment and then adding
> some logic to third-party/gasnet/Makefile that adds the configuration
> options based on CHPL_MAKE_COMM_SUBSTRATE being set to ibv. (?)
IMO large should be used, and the only change should be to define
CHPL_GASNET_SEGMENT as large by default when using ibv.
That can be done by modifiying util/chplenv/commSegment, and changing:
if (($substrate eq "portals") || ($substrate eq "gemini") || ($substrate
eq "aries")) {
$segment = "fast";
} else {
$segment = "everything";
}
To:
if (($substrate eq "portals") || ($substrate eq "gemini") || ($substrate
eq "aries")) {
$segment = "fast";
} elsif ($substrate eq "ibv") {
$segment= "large";
} else {
$segment = "everything";
}
I can do that and commit it if you want.
Greets,
Rafael
> Thanks,
> -Brad
>
> On Fri, 7 Feb 2014, Rafael Larrosa Jiménez wrote:
> > Hi,
> >
> >> Hi Rafael --
> >>
> >> Here's another tip from the GASNet team. Would you be able to give this
> >> a
> >> try and see if it helps with the ibv + fast?
> >>
> >> ----
> >>
> >> The Bad Address sounds like an interaction between IBV and POSIX Shared
> >> Memory:
> >> https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=2929
> >>
> >> The work-around is to configure GASNet with "--enable-pshm-sysv
> >> --disable-pshm-posix".
> >>
> >> ----
> >
> > Yes, it works fine with those changes, sorry for the delay.
> >
> > Greets,
> >
> > Rafael
> >
> >> Thanks,
> >> -Brad
> >>
> >> On Thu, 30 Jan 2014, Brad Chamberlain wrote:
> >>> Hi Rafael --
> >>>
> >>> Thanks for your analysis and description.
> >>>
> >>> I don't have any problem with changing the default segment for 'ibv' and
> >>> agree that it sounds like the right thing to do if it works around this
> >>> issue. It seems like 'fast' would be preferable to 'large' from a
> >>> performance perspective (based on a quick glance at the docs -- I'm
> >>> mostly
> >>> unfamiliar with 'large').
> >>>
> >>> In trying to use the 'fast' segment, do you know whether the memory
> >>> allocator used changed to either tcmalloc or dlmalloc (either
> >>> automatically or manually)? That is probably necessary for correctness,
> >>> but should've (hopefully) happened automatically. printchplenv can be
> >>> used to verify.
> >>>
> >>> Note that I believe you should only need to set CHPL_GASNET_SEGMENT. The
> >>> CHPL_MAKE_ variables should follow automatically (and are not intended
> >>> to
> >>> be set by the end-user).
> >>>
> >>> Thanks again,
> >>> -Brad
> >>>
> >>> On Thu, 30 Jan 2014, Rafael Larrosa Jiménez wrote:
> >>>> El Jueves, 30 de enero de 2014 12:21:11 Akihiro Hayashi escribió:
> >>>>> Hi,
> >>>>>
> >>>>> Thanks for your helping me to understand the problem.
> >>>>> I think now I understand the problem correctly.
> >>>>>
> >>>>> Rafael Asenjo and Rafael Larrosa.
> >>>>> I'm pretty sure my assignment have involved a contiguous block and
> >>>>> useBulkTransfer is active and useBulkTransferStride is not active in
> >>>>> my
> >>>>> chapel compiler. I just wanted to make both sure my code
> >>>>> (useBulkTransfer)
> >>>>> and Rafaerl Larrosa's code (useBulkTransferStride) can fail due to the
> >>>>> ibv-conduit bug
> >>>>> (https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495).
> >>>>
> >>>> First, tell that I'm not the developer, but a client of that code.
> >>>>
> >>>> Second, I have been writting this email for several hours, as I changed
> >>>> my
> >>>> vision on the problem, have been reading the ib verbs manuals, ib code,
> >>>> etc.
> >>>>
> >>>> At the end I have found the solution, by default chapel puts all memory
> >>>> as
> >>>> memory accesible by RDMA, which seems to be the problem, as it can be
> >>>> "unpinned, if instead only "large" portions are used for RDMA then it
> >>>> works
> >>>> fine, as explained in one of the messages:
> >>>>
> >>>> ---
> >>>> In a GASNET_SEGMENT_FAST or _LARGE configuration the segment is
> >>>> obtained
> >>>> at
> >>>> startup via mmap() and is never unmapped. So, sending from inside the
> >>>> GASNet
> >>>> segment in these two cases will ensure this bug cannot occur.
> >>>> ---
> >>>>
> >>>> So if you define :
> >>>>
> >>>> export CHPL_MAKE_GASNET_SEGMENT=large
> >>>> export CHPL_MAKE_COMM_SEGMENT=large
> >>>> export CHPL_GASNET_SEGMENT=large
> >>>>
> >>>> Then do a make clobber followed by a make and recompile your program,
> >>>> it
> >>>> should work fine.
> >>>>
> >>>> BTW, when using portals, gemini or aries, the fast segment is used, as
> >>>> explained in :
> >>>>
> >>>> doc/release/README.multilocale
> >>>> ---
> >>>> 3) Advanced GASNet users can set CHPL_GASNET_SEGMENT to choose a
> >>>>
> >>>> memory segment to use with GASNet. Current defaults are:
> >>>> When CHPL_COMM_SUBSTRATE is... Chapel will choose...
> >>>>
> >>>> portals fast
> >>>> gemini fast
> >>>> (other) everything
> >>>>
> >>>> ---
> >>>>
> >>>> But using fast with ibv gives this error:
> >>>> *** FATAL ERROR: Unexpected error Bad address (rc=1 errno=14) when
> >>>> registering
> >>>> the segment
> >>>>
> >>>> But that is not a problem as large segments are the solution, hope they
> >>>> don't
> >>>> break other things :-)
> >>>>
> >>>> I have tested it with Akihiro code, and it seems to work fine now.
> >>>>
> >>>> Perphaps is a good idea to change the default to use large segments
> >>>> with
> >>>> ibv.
> >>>>
> >>>> Greets,
> >>>>
> >>>> Rafael
> >
> > EMAIL: [email protected] Edificio de Bioinnovación
> > TELEF: + 34951952788 C/ Severo Ochoa 34
> > FAX : +34951952792 Parque Tecnológico de Andalucía
> >
> > 29590 Málaga (SPAIN)
--
Rafael Larrosa Jiménez
Centro de Supercomputación y Bioinformática - http://www.scbi.uma.es
Universidad de Málaga
EMAIL: [email protected] Edificio de Bioinnovación
TELEF: + 34951952788 C/ Severo Ochoa 34
FAX : +34951952792 Parque Tecnológico de Andalucía
29590 Málaga (SPAIN)
------------------------------------------------------------------------------
Android apps run on BlackBerry 10
Introducing the new BlackBerry 10.2.1 Runtime for Android apps.
Now with support for Jelly Bean, Bluetooth, Mapview and more.
Get your Android app in front of a whole new audience. Start now.
http://pubads.g.doubleclick.net/gampad/clk?id=124407151&iu=/4140/ostg.clktrk
_______________________________________________
Chapel-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/chapel-developers
