Testing Hadoop job

2009-08-26 Thread Nikhil Sawant

hi

can u guys suggest some hadoop unit testing framework apart from MRUnit???
i have used MRUnit but i m not sure abt its feasibilty and support to 
hadoop 0.20.
i could not find a proper documentation for MRUnit, is it available 
anywhere?


--
cheers
nikhil



Re: Testing Hadoop job

2009-08-27 Thread Nikhil Sawant

hi
thanks Jason, for prompt reply i will go through "pro hadoop". already 
on my to-do list


any idea abt the MRUnit??? has anyone used it?
i think it is useful as it allows dummy map and reduce drivers which 
accepts (K,V) pair/s and checks the o/p (K,V) pair/s with the expected 
(K,V)..it gives gr8 debugging capabilities while implementing 
complex logics. (all "System.outs" can be seen on console itself!!)

i have used the basic functionalities of MRUnit testing framework
i would like to know the limitations (e.g. i found out that MRUnit does 
not check the partioner logic) and its feasibility with hadoop 0.20...

No proper documentation i found ! :(

cheers
nikhil

Jason Venner wrote:

I put together a framework for the Pro Hadoop book that I use quite a bit,
and has some documentation in the book examples ;)
I haven't tried it with 0.20.0 however.

The nicest thing that I did with the framework was provide a way to run a
persistent mini virtual cluster for running multiple tests on.

On Wed, Aug 26, 2009 at 4:50 PM, Nikhil Sawant wrote:

  

hi

can u guys suggest some hadoop unit testing framework apart from MRUnit???
i have used MRUnit but i m not sure abt its feasibilty and support to
hadoop 0.20.
i could not find a proper documentation for MRUnit, is it available
anywhere?

--
cheers
nikhil






  


No. of records processed by each Reducer

2010-09-22 Thread Nikhil Sawant

 Hi,

Is there any way to get number of records processed by each reducer, 
without using hadoop counters?


Thanks
-Nikhil