I tried this approach, but the job is not distributed among 10 mapper
nodes. Seems Hadoop ignores this property :(
My first thought is, that the small file size is the problem and Hadoop
doesn't care about it's splitting in proper way.
Thanks any ideas.
On 06/16/2012 11:27 AM, Bejoy KS wrote:
Hi Ondrej
You can use NLineInputFormat with n set to 10.
------Original Message------
From: Ondřej Klimpera
To: common-user@hadoop.apache.org
ReplyTo: common-user@hadoop.apache.org
Subject: Setting number of mappers according to number of TextInput lines
Sent: Jun 16, 2012 14:31
Hello,
I have very small input size (kB), but processing to produce some output
takes several minutes. Is there a way how to say, file has 100 lines, i
need 10 mappers, where each mapper node has to process 10 lines of input
file?
Thanks for advice.
Ondrej Klimpera
Regards
Bejoy KS
Sent from handheld, please excuse typos.
