Hi Eitan, Is it possible you have speculative execution enabled? Check to make sure the # of tasks being run matches up with your expectations.
You could also try running the same measurements for TeraGen with different replication factors, for another comparison point. Best, Andrew On Fri, Jan 9, 2015 at 6:32 PM, Eitan Rosenfeld <eita...@gmail.com> wrote: > My goal is to see how the performance and network utilization of TeraSort > is affected by varying the replication factor from 1-3 on my 16-node > cluster. (I have modified TeraSort such that it uses my system's > replication factor.) I am sorting 100GB. > > In particular, I am confused by the network utilization. With 1 replica, > the network utilization is under 1GB. With 2 replicas, it is about 117GB. > And with 3 replicas, it is about 225-230GB. > > I understand that just replicating the 100GB of sorted data causes 100GB > and 200GB of network traffic in the 2 and 3 replica configurations, > respectively. However, what accounts for the extra 17GB and 25-30GB in the > 2 and 3 replica configs? And what accounts for the minimal network usage in > the 1 replica configuration? > > Note that the data is generated with TeraGen using the same replication > factor with which it is later sorted. > > Thank you, > Eitan Rosenfeld >
