You can run a job in local mode without a cluster and run in a debugger enabling breakpoints single stepping - examination of variables ... If you do this 1) Choose a SMALL problem 2) Beware of the fact that there is only one VM - that means be cautions with static variables - they cause leakage from mappers to reducers. 3) Keep testing on the cluster My rule is never try a task on a cluster until it works well in local mode
On Thu, Mar 29, 2012 at 9:13 AM, Wydra, Paul <pwy...@verisign.com> wrote: > You can also decompose your problem so that you have code that is testable > outside the map-reduce process. This way, you can ensure all your "helper" > classes/methods are behaving as you expect, and you can debug this locally. > Once that is done, you are left with only debugging things related to MR > plumbing. MRUnit is good for this as Brock pointed out. But even if you > can't use that for some reason, knowing the helper logic is correct is a > big help with complex jobs. > > -paul > > -----Original Message----- > From: Brock Noland [mailto:br...@cloudera.com] > Sent: Thursday, March 29, 2012 12:08 PM > To: mapreduce-user@hadoop.apache.org > Subject: Re: Debug MR tasks impossible. > > Can I suggest unit testing with MRUnit: > > https://cwiki.apache.org/confluence/display/MRUNIT/Index > https://cwiki.apache.org/confluence/display/MRUNIT/MRUnit+Tutorial > > Brock > > On Thu, Mar 29, 2012 at 10:33 AM, Pedro Costa <psdc1...@gmail.com> wrote: > > Hi, > > > > I'm trying to debug map and reduce tasks for a quite long time, and it > seems > > that it's impossible. MR are launched in new process and there's no way > to > > debug them. Even with IsolationRunner class it's impossible. This isn't > good > > because I really need to debug the class, to understand some changes > that I > > made to the code. > > > > I wonder how MapReduce programmers could debug the code that they > > implemented? > > > > -- > > Best regards, > > > > > > -- > Apache MRUnit - Unit testing MapReduce - > http://incubator.apache.org/mrunit/ > -- Steven M. Lewis PhD 4221 105th Ave NE Kirkland, WA 98033 206-384-1340 (cell) Skype lordjoe_com
