I did try to modify rsocket.c/rstream.c in order to pass to ib_post_send addresses aligned with the page size but I'm not succeeding. Indeed it's not enough to memalign the buffers allocated to page_size but as far as I understood each buffer is sent in chunks this means that those chunks have to be (but last) all in size multiple of page_size as well. I did try increasing RS_OLAP_START_SIZE to page_size (4096) but somehow in the cycle in rsocket.c in rsend the buffer passed to rs_write_data become unaligned. I have not much time right now to follow what is going on in there but I'm sure that Sean can fix it in a blink of an eye.
Let me know, I would like to adopt rsocket in my application instead of my own implementation but until rsocket is not "fixed" I have to stick with my implementation. Gaetano On Wed, Aug 29, 2012 at 6:03 PM, Gaetano Mendola <mend...@gmail.com> wrote: > I was able to make ib_write_bw to suffer of the same NUMA issue, > and this is a good news. > The buffer allocation is aligned to 4K, basically the memory allocation > is done in this way: > > page_size = sysconf(_SC_PAGESIZE); //this is 4K on my system > buf = memalign(page_size, buffer_size); > > with same modification on my code now it doesn't matter on which > NUMA node I do the affinity (mem/cpu) I'm getting around 1400MB/sec > > Gaetano > > > On Wed, Aug 29, 2012 at 10:27 AM, Gaetano Mendola <mend...@gmail.com> wrote: >> On Wed, Aug 29, 2012 at 12:51 AM, Hefty, Sean <sean.he...@intel.com> wrote: >>>> I'm not sure if I have to say sorry for the noise or not but it seems >>>> that the issue was just an NUMA issue! >>> >>> that's good news >>> >>>> May be rsocket has to do a memory affinity on the node with the IB >>>> board attached on >>>> before to allocate his own memory? >>> >>> Do you know what code is necessary for doing this? >> >> It's something I have never done before, apart doing a cpu affinity of >> the current process/thread >> to a particular core, it should not be hard to see what numactl does. >> However before doing >> this, that for me has the taste of an hack we should find out why >> ib_write_bw is not affected by >> cpu/memory affinity. >> >> Gaetano >> >> -- >> cpp-today.blogspot.com > > > > -- > cpp-today.blogspot.com -- cpp-today.blogspot.com -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
