Re: Shuffling over the network for local map data.

2013-01-22 Thread Suresh Kumar
at type of performance tests did you run and what kind of improvements > did you find (or not find)? > > Al > > On Tue, 2013-01-22 at 11:02 -0800, Suresh Kumar wrote: > > I have a patch that tries to use file links instead of making a copy > > of the data that is already ava

Re: Shuffling over the network for local map data.

2013-01-22 Thread Suresh Kumar
ely small > instances, so the issue was not as acute. MAPREDUCE-4049 allows the shuffle > to be pluggable, so you won't have to patch Hadoop framework code itself. > > Are you saying that you don't have access to EC2? > > > On Tue, Jan 22, 2013 at 11:02 AM, Suresh

Re: Shuffling over the network for local map data.

2013-01-22 Thread Suresh Kumar
I have a patch that tries to use file links instead of making a copy of the data that is already available locally. I tested it on the a single machine cluster configuration running 48 mappers and reducers. I unfortunately do not have access to a cluster even a small one. Can some on review and tes

Re: Shuffling over the network for local map data.

2013-01-22 Thread Suresh Kumar
dress. Thanks, Suresh. On Tue, Jan 22, 2013 at 8:46 AM, Steve Loughran wrote: > It's just using the loopback address, right -not going on to the external > network and back again? > > On 22 January 2013 03:22, Suresh Kumar wrote: > > > Hello, > > > > I not