You understood correctly the scenario.
I see your rationale and thanks for your suggestions.
To better explain the problem and my point of view let me make an example.
I want to read two files. In the first one the rows are composed as
the following:
Airport_Id, User_Id, Time
and indicates user
Hi everyone,
how can i know the keys that are associated to a particular reducer in
the setup method?
Let's assume in the setup method to read from a file where each line
is a string that will become a key emitted from mappers.
For each of these lines I would like to know if the string will be a
Hi,
Not sure if I am answering your question, but this is the background. Every
MapReduce job has a partitioner associated to it. The default partitioner
is a HashPartitioner. You can as a user write your own partitioner as well
and plug it into the job. The partitioner is responsible for
Hi Hemanth,
thanks for your reply.
Yes, this partially answered to my question. I know how hash
partitioner works and I guessed something similar.
The piece that I missed was that mapred.task.partition returns the
partition number of the reducer.
So, putting al the pieces together I undersand
Hmm. That feels like a join. Can't you read the input file on the map side
and output those keys along with the original map output keys.. That way
the reducer would automatically get both together ?
On Thu, Mar 28, 2013 at 5:20 PM, Alberto Cordioli
cordioli.albe...@gmail.com wrote:
Hi
Yes, that is a possible solution.
But since the MR job has another scope, the mappers already read other
files (very large) and output tuples.
You cannot control the number of mappers and hence the risk is that a
lot of mappers will be created, and each of them read also the other
file instead of
Hi,
The way I understand your requirement - you have a file that contains a set
of keys. You want to read this file on every reducer and take only those
entries of the set, whose keys correspond to the current reducer.
If the above summary is correct, can I assume that you are potentially
