Re: MR1 next steps

2011-07-27 Thread Steve Loughran
On 27/07/11 06:38, Sharad Agarwal wrote: On Tue, Jul 26, 2011 at 4:52 PM, Steve Loughranste...@apache.org wrote: Why does the MiniMR cluster go away in MR2? As Arun pointed out, MiniMR has lot of JT and TT bits ingrained in it which makes it almost unusable in MR2. I wish tests would have

Re: MR1 next steps

2011-07-27 Thread Sharad Agarwal
On Wed, Jul 27, 2011 at 3:09 PM, Steve Loughran ste...@apache.org wrote: As long as there is an in-vm mini cluster for testing I'm happy. I am building up a list of things to fix in MiniDFS cluster, which is going to stay as is, right? Right. MR2 doesn't change anything on HDFS side. Sharad

Re: MR1 next steps

2011-07-27 Thread Eli Collins
On Tue, Jul 26, 2011 at 10:38 PM, Sharad Agarwal sharad.apa...@gmail.com wrote: On Tue, Jul 26, 2011 at 4:52 PM, Steve Loughran ste...@apache.org wrote: Why does the MiniMR cluster go away in MR2? As Arun pointed out, MiniMR has lot of JT and TT bits ingrained in it which makes it almost

Re: MR1 next steps

2011-07-27 Thread Arun C Murthy
On Jul 27, 2011, at 9:41 AM, Eli Collins wrote: Is the plan to port all the current MR tests to the new Mini*Cluster classes? I don't think a straight port is the right way to go - we should stop relying on the Mini*Clusters for every single test and go the route of unit/mock tests. The

Re: MR1 next steps

2011-07-27 Thread Eli Collins
On Wed, Jul 27, 2011 at 9:53 AM, Arun C Murthy a...@hortonworks.com wrote: On Jul 27, 2011, at 9:41 AM, Eli Collins wrote: Is the plan to port all the current MR tests to the new Mini*Cluster classes? I don't think a straight port is the right way to go - we should stop relying on the

Re: MR1 next steps

2011-07-26 Thread Steve Loughran
On 26/07/11 01:23, Arun C Murthy wrote: On Jul 25, 2011, at 11:59 AM, Eli Collins wrote: The current plan is to remove the MR1 code from trunk after merging in MR-279. One issue is that lots of MR 'unit tests' rely on MiniMRCluster which exposes very minute aspects of the JT/TT -

Re: MR1 next steps

2011-07-26 Thread Sharad Agarwal
On Tue, Jul 26, 2011 at 4:52 PM, Steve Loughran ste...@apache.org wrote: Why does the MiniMR cluster go away in MR2? As Arun pointed out, MiniMR has lot of JT and TT bits ingrained in it which makes it almost unusable in MR2. I wish tests would have used MiniMR via a interface then the

MR1 next steps

2011-07-25 Thread Eli Collins
Hey gang, We've had some discussion on what to do with regard to MR1 when MR2 gets merged in, and wanted to give you a heads up. By MR1 I mean the current MR implementation that uses the JobTracker, TaskTracker, etc. On this thread (http://search-hadoop.com/m/GJliJ1uwjXu) on mapreduce-dev@ we

Re: MR1 next steps

2011-07-25 Thread Allen Wittenauer
On Jul 25, 2011, at 11:59 AM, Eli Collins wrote: Note! MR2 supports the current job API - users don't need to rewrite their jobs to run on MR2 - this is about the MR *implementation* not job compatibility. Note that the move to MR2 will affect some APIs (eg metrics, contrib projects that

Re: MR1 next steps

2011-07-25 Thread Eli Collins
On Mon, Jul 25, 2011 at 3:17 PM, Allen Wittenauer a...@apache.org wrote: On Jul 25, 2011, at 11:59 AM, Eli Collins wrote: Note! MR2 supports the current job API - users don't need to rewrite their jobs to run on MR2 - this is about the MR *implementation* not job compatibility.  Note that the

Re: MR1 next steps

2011-07-25 Thread Eli Collins
On Mon, Jul 25, 2011 at 5:23 PM, Arun C Murthy a...@hortonworks.com wrote: On Jul 25, 2011, at 11:59 AM, Eli Collins wrote: The current plan is to remove the MR1 code from trunk after merging in MR-279. One issue is that lots of MR 'unit tests' rely on MiniMRCluster which exposes very