On Thu, Feb 10, 2011 at 08:39, Edson Ramiro <erlfi...@gmail.com> wrote: > Hi, > > I took a look around on the Internet, but I didn't find any docs about > MiniDFS > and MiniMRCluster. Is there docs about them? > > It remember me this phrase I got from the Herriot [1] page. > "As always your best source of information and knowledge about any software > system is its source code" :)
Yes, this still holds ;) Source code is your best friend for a number of reasons: - this is _the_ best documentation for the code and shows what an application does - it is always up-to-date - developers can focus on their development/testing rather then writing an end-user documents about some internals (which no-one but other developers will ever need) > Do you think is possible to have just one tool to cover all kinds of tests? Sure, why not? I am also a big believer that a single OS would do just fine. > Another question, do you know if is possible to evaluate a MR program, eg > sort, with Herriot considering several test data? Absolutely... Herriot does run work-loads against a physical clusters. So, I don't see why it can be impossible. Would be most effective use of your time? Perhaps not, because Herriot requires a specially tailored (instrumented) cluster to be executed against. What you need, I think, is a simple way to get a jar file containing some tests, drop it to a cluster's gateway machine and run then. Looks like as what we are trying to achieve in iTest I have mentioned earlier. Cos > Thanks in Advance > > -- > Edson Ramiro Lucas Filho > {skype, twitter, gtalk}: erlfilho > http://www.inf.ufpr.br/erlf07/ > > > On Mon, Feb 7, 2011 at 10:29 PM, Konstantin Boudnik <c...@apache.org> wrote: > >> On Mon, Feb 7, 2011 at 04:20, Edson Ramiro <erlfi...@gmail.com> wrote: >> > Well, I'm studying the Hadoop test tools to evaluate some (if there are) >> > deficiences, also trying to compare these tools to see what one cover >> that >> > other doesn't and what is possible to do with each one. >> >> There's also a simulated test cluster infrastructure called MiniDFS >> and MiniMRCluster to allow you to develop functional tests without >> actual cluster deployment. >> >> > As far as I know we have just Herriot and MRUnit for test, and them do >> > different things as you said me :) >> > >> > I'm very interested in your initial version, is there a link? >> >> Not at the moment, but I will send it here as soon as a initial >> version is pushed out. >> >> > >> > Thanks in advance >> > >> > -- >> > Edson Ramiro Lucas Filho >> > {skype, twitter, gtalk}: erlfilho >> > http://www.inf.ufpr.br/erlf07/ >> > >> > >> > On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <c...@apache.org> >> wrote: >> > >> >> Yes, Herriot can be used for integration tests of MR. Unit test is a >> very >> >> different thing and normally is done against a 'unit of compilation' >> e.g. a >> >> class, etc. Typically you won't expect to do unit tests against a >> deployed >> >> cluster. >> >> >> >> There is fault injection framework wich works at the level of functional >> >> tests >> >> (with mini-clusters). Shortly we'll be opening an initial version of >> smoke >> >> and >> >> integration test framework (maven and JUnit based). >> >> >> >> It'd be easier to provide you with a hint if you care to explain what >> >> you're >> >> trying to solve. >> >> >> >> Cos >> >> >> >> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote: >> >> > Thank you a lot Konstantin, you cleared my mind. >> >> > >> >> > So, Herriot is a framework designed to test Hadoop as a whole, and >> (IMHO) >> >> is >> >> > a tool for help Hadoop developers and not for who is developing MR >> >> programs, >> >> > but can we use Herriot to do unit, integration or other tests on our >> MR >> >> > jobs? >> >> > >> >> > Do you know another test tool or test framework for Hadoop? >> >> > >> >> > Thanks in Advance >> >> > >> >> > -- >> >> > Edson Ramiro Lucas Filho >> >> > {skype, twitter, gtalk}: erlfilho >> >> > http://www.inf.ufpr.br/erlf07/ >> >> > >> >> > >> >> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <c...@apache.org> >> >> wrote: >> >> > >> >> > > (Moving to common-user where this belongs) >> >> > > >> >> > > Herriot is system test framework which runs against a real physical >> >> > > cluster deployed with a specially crafted build of Hadoop. That >> >> > > instrumented build of provides an extra APIs not available in Hadoop >> >> > > otherwise. These APIs are created to facilitate cluster software >> >> > > testability. Herriot isn't limited by MR but also covered (although >> in >> >> > > a somewhat lesser extend) HDFS side of Hadoop. >> >> > > >> >> > > MRunit is for MR job "unit" testing as in making sure that your MR >> job >> >> > > is ok and/or to allow you to debug it locally before scale >> deployment. >> >> > > >> >> > > So, long story short - they are very different ;) Herriot can do >> >> > > intricate fault injection and can work closely with a deployed >> cluster >> >> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs >> >> > > testing. >> >> > > >> >> > > Hope it helps. >> >> > > -- >> >> > > Take care, >> >> > > Konstantin (Cos) Boudnik >> >> > > >> >> > > >> >> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <erlfi...@gmail.com> >> wrote: >> >> > > > Hi all, >> >> > > > >> >> > > > Plz, could you explain me the difference between MRUnit and >> Herriot? >> >> > > > >> >> > > > I've read the documentation of both and they seem very similar to >> me. >> >> > > > >> >> > > > Is Herriot an evolution of MRUnit? >> >> > > > >> >> > > > What can Herriot do that MRUnit can't? >> >> > > > >> >> > > > Thanks in Advance >> >> > > > >> >> > > > -- >> >> > > > Edson Ramiro Lucas Filho >> >> > > > {skype, twitter, gtalk}: erlfilho >> >> > > > http://www.inf.ufpr.br/erlf07/ >> >> > > > >> >> > > >> >> >> >> -----BEGIN PGP SIGNATURE----- >> >> Version: GnuPG v1.4.10 (GNU/Linux) >> >> >> >> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe >> >> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD >> >> =h/66 >> >> -----END PGP SIGNATURE----- >> >> >> >> >> > >> >