Konstantions, A simple way is to rewrite MPI_Bcast() and insert timer and PMPI_Barrier() before invoking the real PMPI_Bcast(). time spent in PMPI_Barrier() can be seen as time NOT spent on actual data transmission, and since all tasks are synchronized upon exit, time spent in PMPI_Bcast() can be seen as time spent on actual data transmission. this is not perfect, but this is a pretty good approximation. You can add extra timers so you end up with an idea of how much time is spent in PMPI_Barrier() vs PMPI_Bcast().
Cheers, Gilles On Mon, Oct 23, 2017 at 4:16 PM, Konstantinos Konstantinidis <kostas1...@gmail.com> wrote: > In any case, do you think that the time NOT spent on actual data > transmission can impact the total time of the broadcast especially when > there are so many groups that communicate (please refer to the numbers I > gave before if you want to get an idea). > > Also, is there any way to quantify this impact i.e. to measure the time not > spent on actual data transmissions? > > Kostas > > On Fri, Oct 20, 2017 at 10:32 PM, Jeff Hammond <jeff.scie...@gmail.com> > wrote: >> >> Broadcast is collective but not necessarily synchronous in the sense you >> imply. If you broadcast message size under the eager limit, the root may >> return before any non-root processes enter the function. Data transfer may >> happen prior to processes entering the function. Only rendezvous forces >> synchronization between any two processes but there may still be asynchrony >> between different levels of the broadcast tree. >> >> Jeff >> >> On Fri, Oct 20, 2017 at 3:27 PM Konstantinos Konstantinidis >> <kostas1...@gmail.com> wrote: >>> >>> Hi, >>> >>> I am running some tests on Amazon EC2 and they require a lot of >>> communication among m3.large instances. >>> >>> I would like to give you an idea of what kind of communication takes >>> place. There are 40 m3.large instances. Now, 28672 groups of 5 instances are >>> formed in a specific manner (let's skip the details). Within each group, >>> each instance broadcasts some unsigned char data to the other 4 instances in >>> the group. So within each group, exactly 5 broadcasts take place. >>> >>> The problem is that if I increase the size of the group from 5 to 10 >>> there is significant delay in terms of transmission rate while, based on >>> some theoretical results, this is not reasonable. >>> >>> I want to check if one of the reasons that this is happening is due to >>> the time needed for the instances to synchronize when they call MPI_Bcast() >>> since it's a collective function. As far as I know, all of the machines in >>> the broadcast need to call it and then synchronize until the actual data >>> transfer starts. Is there any way to measure this synchronization time? >>> >>> The code is in C++ and the MPI installed is described in the attached >>> file. >>> _______________________________________________ >>> users mailing list >>> users@lists.open-mpi.org >>> https://lists.open-mpi.org/mailman/listinfo/users >> >> -- >> Jeff Hammond >> jeff.scie...@gmail.com >> http://jeffhammond.github.io/ > > > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users
