)
>
> The experiment was done in a cloud hosted environment running set of VMs.
>
> ~Brad
>
> -Original Message-
> From: Prashant Kommireddi [mailto:prash1...@gmail.com]
> Sent: Tuesday, December 13, 2011 9:46 AM
> To: common-user@hadoop.apache.org
> Subject:
can go wrong
and how to fix it.
If I get time, I'll have to find my copy of "Up Front" by Bill Maudlin. There's
a cartoon that really fits you.
Later
> To: common-user@hadoop.apache.org
> Subject: RE: More cores Vs More Nodes ?
> From: tdeut...@us.ibm.com
&
apache.org
Subject: Re: More cores Vs More Nodes ?
You're using OS virtualization in your test. Are you using it in production?
Russell Jurney
twitter.com/rjurney
russell.jur...@gmail.com
datasyndrome.com
On Dec 13, 2011, at 5:16 PM, Brad Sarsfield wrote:
> The experiment was done
gt;
> ~Brad
>
> -Original Message-
> From: Prashant Kommireddi [mailto:prash1...@gmail.com]
> Sent: Tuesday, December 13, 2011 9:46 AM
> To: common-user@hadoop.apache.org
> Subject: Re: More cores Vs More Nodes ?
>
> Hi Brad, how many taskstrackers did you have
Your eagerness to insult is throwing you off track here Michael.
For example, the workload profile of a cluster doing heavy NLP is very
different than one doing serving as a destination for large scale
application/web logs. Ditto for P&C risk modeling vs smart meter use
cases, etc etc...Those
that, you really, really don't want to build a custom cluster for
>a specific job unless you know that you're going to be running that
>specific job or set of jobs (24x7X365) [And yes, I came across such a use
>case...]
>
>HTH
>
>-Mike
>> From: bbock...@cse.unl
ernet myths and 'truisms'.
Unless you've had your hands dirty and at customer's sites you're going to find
the real world is a different place. ;-)
But hey! What do I know?
> To: common-user@hadoop.apache.org
> Subject: RE: More cores Vs More Nodes ?
> From: tdeu
o build a custom cluster for a
specific job unless you know that you're going to be running that specific job
or set of jobs (24x7X365) [And yes, I came across such a use case...]
HTH
-Mike
> From: bbock...@cse.unl.edu
> Subject: Re: More cores Vs More Nodes ?
> Date: Wed,
Putting aside any smarmy responses for a moment - sorry that "job(s)"
wasn't understood as equating to "purpose".
If you are building a general purpose sandbox then I think we all agree on
building a "balanced" general purpose cluster. But if you have production
use cases in mind then you darn
your cluster design are more along the
> lines of the purpose of the cluster and then the budget along with your IT
> constraints.
>
> IMHO its better to avoid building purpose built clusters. You end up not
> being able to easily recycle the hardware in to new clusters easily.
mon-user@hadoop.apache.org
> Subject: RE: More cores Vs More Nodes ?
> From: tdeut...@us.ibm.com
> Date: Tue, 13 Dec 2011 09:46:49 -0800
>
> It also helps to know the profile of your job in how you spec the
> machines. So in addition to Brad's response you should consider if you
> th
ocusing on only one variable.
This is not to say that its a bad question because it forces you to realize
that there are definitely lots of other options. that you have to consider.
HTH
-Mike
> Date: Tue, 13 Dec 2011 20:25:17 -0600
> Subject: Re: More cores Vs More Nodes ?
> Fro
ng set of VMs.
>
> ~Brad
>
> -Original Message-
> From: Prashant Kommireddi [mailto:prash1...@gmail.com]
> Sent: Tuesday, December 13, 2011 9:46 AM
> To: common-user@hadoop.apache.org
> Subject: Re: More cores Vs More Nodes ?
>
> Hi Brad, how many taskstrackers
running set of VMs.
~Brad
-Original Message-
From: Prashant Kommireddi [mailto:prash1...@gmail.com]
Sent: Tuesday, December 13, 2011 9:46 AM
To: common-user@hadoop.apache.org
Subject: Re: More cores Vs More Nodes ?
Hi Brad, how many taskstrackers did you have on each node in both cases
ibm.com
>
>
>
>
> Brad Sarsfield
> 12/13/2011 09:41 AM
> Please respond to
> common-user@hadoop.apache.org
>
>
> To
> "common-user@hadoop.apache.org"
> cc
>
> Subject
> RE: More cores Vs More Nodes ?
>
>
>
>
>
>
> Praveen
more nodes means more IO on read on mapper step
If you use combiners you might need to send only small amount of data over
network to reducers
Alexander
On Tue, Dec 13, 2011 at 12:45 PM, real great.. wrote:
> more cores might help in hadoop environments as there would be more data
> locality.
more cores might help in hadoop environments as there would be more data
locality.
your thoughts?
On Tue, Dec 13, 2011 at 11:11 PM, Brad Sarsfield wrote:
> Praveenesh,
>
> Your question is not naïve; in fact, optimal hardware design can
> ultimately be a very difficult question to answer on what
ation Management
Big Data Technologies
IBM
3565 Harbor Blvd
Costa Mesa, CA 92626-1420
tdeut...@us.ibm.com
Brad Sarsfield
12/13/2011 09:41 AM
Please respond to
common-user@hadoop.apache.org
To
"common-user@hadoop.apache.org"
cc
Subject
RE: More cores Vs More Nodes ?
Praveenesh,
Your
Hi Brad, how many taskstrackers did you have on each node in both cases?
Thanks,
Prashant
Sent from my iPhone
On Dec 13, 2011, at 9:42 AM, Brad Sarsfield wrote:
> Praveenesh,
>
> Your question is not naïve; in fact, optimal hardware design can ultimately
> be a very difficult question to answ
Praveenesh,
Your question is not naïve; in fact, optimal hardware design can ultimately be
a very difficult question to answer on what would be "better". If you made me
pick one without much information I'd go for more machines. But...
It all depends; and there is no right answer :)
Mo
20 matches
Mail list logo
