Re: Reading partition for reducer
The question should be more specific here: Do you want to process a map's sorted total output or do you want to pre-process a whole partition (i.e. all data pertaining to one reducer)? Former would be more ideal inside MapTask.java, latter in ReduceTask.java. On Mon, Apr 1, 2013 at 5:36 PM, Vikas Jadhav vikascjadha...@gmail.com wrote: Hello I want to process output of mapper to processed before it is sent to reducer. @ what point i should hook in my code processing i guess it is ReduceTask.java file if anyone knows reagarding this please help me in this. Thank You. -- * * * Thanx and Regards* * Vikas Jadhav* -- Harsh J
Re: Reading partition for reducer
Hi I want process/store all data pertaining to one reducer. i want store it in some data structure depending on key for example (0,ABC) (0,TER) (1,DEF) (1,XYZ) for key 0 and 1 data will be in different data structure. then perform cross product of above datasets Currently i am looking in ReduceTask.java Thank You. On Mon, Apr 1, 2013 at 7:21 PM, Harsh J ha...@cloudera.com wrote: The question should be more specific here: Do you want to process a map's sorted total output or do you want to pre-process a whole partition (i.e. all data pertaining to one reducer)? Former would be more ideal inside MapTask.java, latter in ReduceTask.java. On Mon, Apr 1, 2013 at 5:36 PM, Vikas Jadhav vikascjadha...@gmail.com wrote: Hello I want to process output of mapper to processed before it is sent to reducer. @ what point i should hook in my code processing i guess it is ReduceTask.java file if anyone knows reagarding this please help me in this. Thank You. -- * * * Thanx and Regards* * Vikas Jadhav* -- Harsh J -- * * * Thanx and Regards* * Vikas Jadhav*
Re: Reading partition for reducer
have you considered joining the u...@hadoop.apache.org and asking the question there? On 1 April 2013 17:38, Vikas Jadhav vikascjadha...@gmail.com wrote: Hi I want process/store all data pertaining to one reducer. i want store it in some data structure depending on key for example (0,ABC) (0,TER) (1,DEF) (1,XYZ) for key 0 and 1 data will be in different data structure. then perform cross product of above datasets Currently i am looking in ReduceTask.java Thank You. On Mon, Apr 1, 2013 at 7:21 PM, Harsh J ha...@cloudera.com wrote: The question should be more specific here: Do you want to process a map's sorted total output or do you want to pre-process a whole partition (i.e. all data pertaining to one reducer)? Former would be more ideal inside MapTask.java, latter in ReduceTask.java. On Mon, Apr 1, 2013 at 5:36 PM, Vikas Jadhav vikascjadha...@gmail.com wrote: Hello I want to process output of mapper to processed before it is sent to reducer. @ what point i should hook in my code processing i guess it is ReduceTask.java file if anyone knows reagarding this please help me in this. Thank You. -- * * * Thanx and Regards* * Vikas Jadhav* -- Harsh J -- * * * Thanx and Regards* * Vikas Jadhav*
Re: Reading partition for reducer
yes i have joined u...@hadoop.apache.org mailing list. i think it is not possible to do in user code(map or reduce function) rather i dont want to make my changes visible to mapreduce programmer thats why i thought it is good idea to ask question here in this mailing list. On Tue, Apr 2, 2013 at 2:29 AM, Steve Loughran ste...@hortonworks.comwrote: have you considered joining the u...@hadoop.apache.org and asking the question there? On 1 April 2013 17:38, Vikas Jadhav vikascjadha...@gmail.com wrote: Hi I want process/store all data pertaining to one reducer. i want store it in some data structure depending on key for example (0,ABC) (0,TER) (1,DEF) (1,XYZ) for key 0 and 1 data will be in different data structure. then perform cross product of above datasets Currently i am looking in ReduceTask.java Thank You. On Mon, Apr 1, 2013 at 7:21 PM, Harsh J ha...@cloudera.com wrote: The question should be more specific here: Do you want to process a map's sorted total output or do you want to pre-process a whole partition (i.e. all data pertaining to one reducer)? Former would be more ideal inside MapTask.java, latter in ReduceTask.java. On Mon, Apr 1, 2013 at 5:36 PM, Vikas Jadhav vikascjadha...@gmail.com wrote: Hello I want to process output of mapper to processed before it is sent to reducer. @ what point i should hook in my code processing i guess it is ReduceTask.java file if anyone knows reagarding this please help me in this. Thank You. -- * * * Thanx and Regards* * Vikas Jadhav* -- Harsh J -- * * * Thanx and Regards* * Vikas Jadhav* -- * * * Thanx and Regards* * Vikas Jadhav*
