Regarding your original question, you can use the min and max split settings to control the number of maps: http://hadoop.apache.org/docs/stable/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html . See #setMinInputSplitSize and #setMaxInputSplitSize. Or use mapred.min.split.size directly.
W.r.t your custom inputformat, are you sure you job is using this InputFormat and not the default one? HTH, +Vinod Kumar Vavilapalli Hortonworks Inc. http://hortonworks.com/ On Jan 28, 2013, at 12:56 PM, Marcelo Elias Del Valle wrote: > Just to complement the last question, I have implemented the getSplits method > in my input format: > https://github.com/mvallebr/CSVInputFormat/blob/master/src/main/java/org/apache/hadoop/mapreduce/lib/input/CSVNLineInputFormat.java > > However, it still doesn't create more than 2 map tasks. Is there something I > could do about it to assure more map tasks are created? > > Thanks > Marcelo. > > > 2013/1/28 Marcelo Elias Del Valle <mvall...@gmail.com> > Sorry for asking too many questions, but the answers are really happening. > > > 2013/1/28 Harsh J <ha...@cloudera.com> > This seems CPU-oriented. You probably want the NLineInputFormat? See > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/lib/NLineInputFormat.html. > This should let you spawn more maps as we, based on your N factor. > > Indeed, CPU is my bottleneck. That's why I want more things in parallel. > Actually, I wrote my own InputFormat, to be able to process multiline CSVs: > https://github.com/mvallebr/CSVInputFormat > I could change it to read several lines at a time, but would this alone allow > more tasks running in parallel? > > Not really - "Slots" are capacities, rather than split factors > themselves. You can have N slots always available, but your job has to > supply as many map tasks (based on its input/needs/etc.) to use them > up. > > But how can I do that (supply map tasks) in my job? changing its code? hadoop > config? > > Unless your job sets the number of reducers to 0 manually, 1 default > reducer is always run that waits to see if it has any outputs from > maps. If it does not receive any outputs after maps have all > completed, it dies out with behavior equivalent to a NOP. > Ok, I did job.setNumReduceTasks(0); , guess this will solve this part, thanks! > > > -- > Marcelo Elias Del Valle > http://mvalle.com - @mvallebr > > > > -- > Marcelo Elias Del Valle > http://mvalle.com - @mvallebr
