The problem is that I profiled the receiving node and found that its network bandwidth is used only less than 50%. That's why I want to find ways to increase the receiving throughput. Any ideas ?
Lei -----邮件原件----- 发件人: users [mailto:users-boun...@open-mpi.org] 代表 George Bosilca 发送时间: 2014年8月27日 20:39 收件人: Open MPI Users 主题: Re: [OMPI users] 答复: Does multiple Irecv means concurrent receiving ? You have a physical constraint, the capacity of your links. If you are over 90% of your network bandwidth, there is little to be improved. George. On Aug 27, 2014, at 0:18, "Zhang,Lei(Ecom)" <zhangle...@baidu.com> wrote: >> I'm not sure what you mean by this statement. If you add N asynchronous >> requests and the speed is not decreased, that's a *good* thing, right? > > My problem is that N asynchronous irecv does not *increase* the speed of > receiving data compared to just 1 irecv. > I have multiple nodes sending large messages to the receiving node. But it > seems that the multiple irecv are not overlapped. > The delay between the completion of different irecvs is the same as what I > get when there is only 1 irecv. > > I suspect that maybe sending many large messages to one node at (almost) the > same time causes network congestion. > But I did not improve receive speed even if I reduce the network congestion > by sending large messages at different times. > > I know the problem depends on many things, but is eager to know how to fix it > to improve the receiving speed. > > Lei > > -----邮件原件----- > 发件人: users [mailto:users-boun...@open-mpi.org] 代表 Jeff Squyres > (jsquyres) > 发送时间: 2014年8月23日 20:03 > 收件人: Open MPI User's List > 主题: Re: [OMPI users] Does multiple Irecv means concurrent receiving ? > >> On Aug 20, 2014, at 3:37 AM, Zhang,Lei(Ecom) <zhangle...@baidu.com> wrote: >> >> I have a performance problem with receiving. In a single master thread, I >> made several Irecv calls: >> >> Irecv(buf1, ..., tag, ANY_SOURCE, COMM_WORLD) Irecv(buf2, ..., tag, >> ANY_SOURCE, COMM_WORLD) ... >> Irecv(bufn, ..., tag, ANY_SOURCE, COMM_WORLD) >> >> all of which try to receive from any node for messages with the same tag. >> >> Then, whenever any of the Irecv completes (using Testany), a separate thread >> is dispatched to work on the received message. >> In my program, many nodes will send to this master thread. >> >> However, I noticed that the speed of recv is almost unaffected no matter how >> many Irecv calls were made. > > I'm not sure what you mean by this statement. If you add N asynchronous > requests and the speed is not decreased, that's a *good* thing, right? > >> It seems that multiple Irecv calls does not mean concurrently receiving from >> many nodes. > > That's a different question. > > It depends on many factors, including your underlying network software and > hardware. If you post many long message Irecvs, for example, and multiple > senders concurrently send matching messages, there will almost certainly be > some overlap in the receive of messages between the different senders. > > Is that what you are asking? > >> By profiling the node running the master thread, I can see that the network >> input bandwidth is quite low. > > That could also be for many reasons. Are you sending short messages? Are > you sending messages infrequently? Is your network otherwise congested? > ...(etc.)? > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/08/25140.php > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/08/25158.php _______________________________________________ users mailing list us...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2014/08/25164.php
