On Thu, Feb 10, 2011 at 08:39, Edson Ramiro <erlfi...@gmail.com> wrote:
> Hi,
>
> I took a look around on the Internet, but I didn't find any docs about
> MiniDFS
> and MiniMRCluster. Is there docs about them?
>
> It remember me this phrase I got from the Herriot [1] page.
> "As always your best source of information and knowledge about any software
> system is its source code" :)
Yes, this still holds ;) Source code is your best friend for a number
of reasons:
- this is _the_ best documentation for the code and shows what an
application does
- it is always up-to-date
- developers can focus on their development/testing rather then
writing an end-user documents about some internals (which no-one but
other developers will ever need)
> Do you think is possible to have just one tool to cover all kinds of tests?
Sure, why not? I am also a big believer that a single OS would do just fine.
> Another question, do you know if is possible to evaluate a MR program, eg
> sort, with Herriot considering several test data?
Absolutely... Herriot does run work-loads against a physical clusters.
So, I don't see why it can be impossible. Would be most effective use
of your time? Perhaps not, because Herriot requires a specially
tailored (instrumented) cluster to be executed against.
What you need, I think, is a simple way to get a jar file containing
some tests, drop it to a cluster's gateway machine and run then. Looks
like as what we are trying to achieve in iTest I have mentioned
earlier.
Cos
> Thanks in Advance
>
> --
> Edson Ramiro Lucas Filho
> {skype, twitter, gtalk}: erlfilho
> http://www.inf.ufpr.br/erlf07/
>
>
> On Mon, Feb 7, 2011 at 10:29 PM, Konstantin Boudnik <c...@apache.org> wrote:
>
>> On Mon, Feb 7, 2011 at 04:20, Edson Ramiro <erlfi...@gmail.com> wrote:
>> > Well, I'm studying the Hadoop test tools to evaluate some (if there are)
>> > deficiences, also trying to compare these tools to see what one cover
>> that
>> > other doesn't and what is possible to do with each one.
>>
>> There's also a simulated test cluster infrastructure called MiniDFS
>> and MiniMRCluster to allow you to develop functional tests without
>> actual cluster deployment.
>>
>> > As far as I know we have just Herriot and MRUnit for test, and them do
>> > different things as you said me :)
>> >
>> > I'm very interested in your initial version, is there a link?
>>
>> Not at the moment, but I will send it here as soon as a initial
>> version is pushed out.
>>
>> >
>> > Thanks in advance
>> >
>> > --
>> > Edson Ramiro Lucas Filho
>> > {skype, twitter, gtalk}: erlfilho
>> > http://www.inf.ufpr.br/erlf07/
>> >
>> >
>> > On Fri, Feb 4, 2011 at 3:40 AM, Konstantin Boudnik <c...@apache.org>
>> wrote:
>> >
>> >> Yes, Herriot can be used for integration tests of MR. Unit test is a
>> very
>> >> different thing and normally is done against a 'unit of compilation'
>> e.g. a
>> >> class, etc. Typically you won't expect to do unit tests against a
>> deployed
>> >> cluster.
>> >>
>> >> There is fault injection framework wich works at the level of functional
>> >> tests
>> >> (with mini-clusters). Shortly we'll be opening an initial version of
>> smoke
>> >> and
>> >> integration test framework (maven and JUnit based).
>> >>
>> >> It'd be easier to provide you with a hint if you care to explain what
>> >> you're
>> >> trying to solve.
>> >>
>> >> Cos
>> >>
>> >> On Thu, Feb 03, 2011 at 10:25AM, Edson Ramiro wrote:
>> >> > Thank you a lot Konstantin, you cleared my mind.
>> >> >
>> >> > So, Herriot is a framework designed to test Hadoop as a whole, and
>> (IMHO)
>> >> is
>> >> > a tool for help Hadoop developers and not for who is developing MR
>> >> programs,
>> >> > but can we use Herriot to do unit, integration or other tests on our
>> MR
>> >> > jobs?
>> >> >
>> >> > Do you know another test tool or test framework for Hadoop?
>> >> >
>> >> > Thanks in Advance
>> >> >
>> >> > --
>> >> > Edson Ramiro Lucas Filho
>> >> > {skype, twitter, gtalk}: erlfilho
>> >> > http://www.inf.ufpr.br/erlf07/
>> >> >
>> >> >
>> >> > On Wed, Feb 2, 2011 at 4:58 PM, Konstantin Boudnik <c...@apache.org>
>> >> wrote:
>> >> >
>> >> > > (Moving to common-user where this belongs)
>> >> > >
>> >> > > Herriot is system test framework which runs against a real physical
>> >> > > cluster deployed with a specially crafted build of Hadoop. That
>> >> > > instrumented build of provides an extra APIs not available in Hadoop
>> >> > > otherwise. These APIs are created to facilitate cluster software
>> >> > > testability. Herriot isn't limited by MR but also covered (although
>> in
>> >> > > a somewhat lesser extend) HDFS side of Hadoop.
>> >> > >
>> >> > > MRunit is for MR job "unit" testing as in making sure that your MR
>> job
>> >> > > is ok and/or to allow you to debug it locally before scale
>> deployment.
>> >> > >
>> >> > > So, long story short - they are very different ;) Herriot can do
>> >> > > intricate fault injection and can work closely with a deployed
>> cluster
>> >> > > (say control Hadoop nodes and daemons); MRUnit is focused on MR jobs
>> >> > > testing.
>> >> > >
>> >> > > Hope it helps.
>> >> > > --
>> >> > > Take care,
>> >> > > Konstantin (Cos) Boudnik
>> >> > >
>> >> > >
>> >> > > On Wed, Feb 2, 2011 at 05:44, Edson Ramiro <erlfi...@gmail.com>
>> wrote:
>> >> > > > Hi all,
>> >> > > >
>> >> > > > Plz, could you explain me the difference between MRUnit and
>> Herriot?
>> >> > > >
>> >> > > > I've read the documentation of both and they seem very similar to
>> me.
>> >> > > >
>> >> > > > Is Herriot an evolution of MRUnit?
>> >> > > >
>> >> > > > What can Herriot do that MRUnit can't?
>> >> > > >
>> >> > > > Thanks in Advance
>> >> > > >
>> >> > > > --
>> >> > > > Edson Ramiro Lucas Filho
>> >> > > > {skype, twitter, gtalk}: erlfilho
>> >> > > > http://www.inf.ufpr.br/erlf07/
>> >> > > >
>> >> > >
>> >>
>> >> -----BEGIN PGP SIGNATURE-----
>> >> Version: GnuPG v1.4.10 (GNU/Linux)
>> >>
>> >> iF4EAREIAAYFAk1LkUYACgkQenyFlstYjhIyYwD9HM7YvfdcvBuqdN24No5T4dLe
>> >> lDLVlnEs8QIN4V7RqAYBAJ8liUG2YZ+c/wvWL3/lVAGY+Fqls0k4OYLG4rXJrwwD
>> >> =h/66
>> >> -----END PGP SIGNATURE-----
>> >>
>> >>
>> >
>>
>
