reducers and data locality

Hello folks, I have a lot of input splits (10k-50k - 128 mb blocks) which contains text files. I need to process those line by line, then copy the result into roughly equal size of shards. So i generate a random key (from a range of [0:numberOfShards]) which is used to route the map output to

Re: reducers and data locality

To: common-user@hadoop.apache.org Reply-To: common-user@hadoop.apache.org Subject: reducers and data locality Hello folks, I have a lot of input splits (10k-50k - 128 mb blocks) which contains text files. I need to process those line by line, then copy the result into roughly equal size of shards