Re: Streaming jar creates only 1 reducer

You can also use -numReduceTasks <#reduces> option to streaming. On Oct 21, 2011, at 10:22 PM, Mapred Learn wrote: > Thanks Harsh ! > This is exactly what I thought ! > > And don't know what you mean by cross-post ? I just posted to mapred and HDFS > mailing lists ? What's your point about cros

Re: Streaming jar creates only 1 reducer

Mapred, Glad to know you are able to control your # of reducers now. I believe you answered yourself there :) Doing http://en.wikipedia.org/wiki/Crossposting often leads to nose, confusion and duplicated response efforts on the lists. On Sat, Oct 22, 2011 at 10:52 AM, Mapred Learn wrote: > Tha

Re: Streaming jar creates only 1 reducer

Thanks Harsh ! This is exactly what I thought ! And don't know what you mean by cross-post ? I just posted to mapred and HDFS mailing lists ? What's your point about cross-pointing ?? Sent from my iPhone On Oct 21, 2011, at 8:57 PM, Harsh J wrote: > Mapred, > > You need to pass -Dmapred.redu

Re: Streaming jar creates only 1 reducer

Mapred, You need to pass -Dmapred.reduce.tasks=N along. Reducers are a per-job configurable number, unlike mappers whose numbers can be determined based on inputs. P.s. Please do not cross post questions to multiple lists. On 22-Oct-2011, at 4:05 AM, Mapred Learn wrote: > Do you know what par

Re: Streaming jar creates only 1 reducer

Do you know what parameters from conf files ? Thanks, Sent from my iPhone On Oct 21, 2011, at 3:32 PM, Nick Jones wrote: > FWIW, I usually specify the number of reducers in both streaming and > against the Java API. The "default" is what's read from your config > files on the submitting node.

Re: Streaming jar creates only 1 reducer

FWIW, I usually specify the number of reducers in both streaming and against the Java API. The "default" is what's read from your config files on the submitting node. Nick Jones On Oct 21, 2011, at 5:00 PM, Mapred Learn wrote: > Hi, > Does streaming jar create 1 reducer by default ? We have red

Streaming jar creates only 1 reducer

Hi, Does streaming jar create 1 reducer by default ? We have reduce tasks per task tracker configured to be more than 1 but my job has about 150 mappers and only 1 reducer: reducer.py basically just reads the line and prints it. Why doesn't streaming.jar invokes multiple reducers for this case ?