: common-user@hadoop.apache.org
Subject: RE: Sanity check re: value of 10GbE NICs for Hadoop?
Matt,
Thanks, this is helpful, I was wondering if you may have some thoughts
on the list of other potential benefits of 10GbE NICs for Hadoop
(listed in my original e-mail to the list)?
regards,
Saqib
On Wed, Jun 29, 2011 at 12:27 AM, Matt Davies wrote:
> ... I've seen that our cluster
> could use more bandwidth, but it wasn't to the nodes that made the big
> difference, it was getting better switches that had better backplanes - the
> fabric made the difference.
>
Any recommendations on spec
e 10GBe? I would plan only if your data is showing the network
saturation.
-Bharath
From: Michel Segel
To: "common-user@hadoop.apache.org"
Sent: Wednesday, June 29, 2011 2:04 PM
Subject: Re: Sanity check re: value of 10GbE NICs for Hadoop?
I&#x
original e-mail to the list)?
>
> regards,
> Saqib
>
> -Original Message-
> From: Matthew Foley [mailto:ma...@yahoo-inc.com]
> Sent: Tuesday, June 28, 2011 12:04 PM
> To: common-user@hadoop.apache.org
> Cc: Matthew Foley
> Subject: Re: Sanity check re: value o
original e-mail to the list)?
>
> regards,
> Saqib
>
> -Original Message-
> From: Matthew Foley [mailto:ma...@yahoo-inc.com]
> Sent: Tuesday, June 28, 2011 12:04 PM
> To: common-user@hadoop.apache.org
> Cc: Matthew Foley
> Subject: Re: Sanity check re: value o
l continue. These systems can
> saturate the 1Gb NICs.
>
>
> -Bharath
>
>
>
>
> From: Saqib Jang -- Margalla Communications
> To: common-user@hadoop.apache.org
> Sent: Tuesday, June 28, 2011 10:16 AM
> Subject: Sanity check re: value o
3x12TB disks. This is the trend now and will continue. These systems can
saturate the 1Gb NICs.
-Bharath
From: Saqib Jang -- Margalla Communications
To: common-user@hadoop.apache.org
Sent: Tuesday, June 28, 2011 10:16 AM
Subject: Sanity check re: value of
I would say this is quite a difficult choice. I've seen that our cluster
could use more bandwidth, but it wasn't to the nodes that made the big
difference, it was getting better switches that had better backplanes - the
fabric made the difference.
I've also seen some workloads where job design is
Price the cost of 1GbE->10GbE vs. more nodes, using data from monitoring
your cluster during peak load. It should be clear which is a better value.
Russ
On Tue, Jun 28, 2011 at 4:05 PM, Mathias Herberts <
mathias.herbe...@gmail.com> wrote:
> On Wed, Jun 29, 2011 at 01:02, Matei Zaharia
> wrote
On Wed, Jun 29, 2011 at 01:02, Matei Zaharia wrote:
> Ideally, to evaluate whether you want to go for 10GbE NICs, you would profile
> your target Hadoop workload and see whether it's communication-bound. Hadoop
> jobs can definitely be communication-bound if you shuffle a lot of data
> between
tial benefits of 10GbE NICs for Hadoop
>> (listed in my original e-mail to the list)?
>>
>> regards,
>> Saqib
>>
>> -Original Message-
>> From: Matthew Foley [mailto:ma...@yahoo-inc.com]
>> Sent: Tuesday, June 28, 2011 12:04 PM
>>
y, June 28, 2011 12:04 PM
> To: common-user@hadoop.apache.org
> Cc: Matthew Foley
> Subject: Re: Sanity check re: value of 10GbE NICs for Hadoop?
>
> Hadoop common provides an abstract FileSystem class, and Hadoop applications
> should be designed to run on that. HDFS is just on
, June 28, 2011 12:04 PM
To: common-user@hadoop.apache.org
Cc: Matthew Foley
Subject: Re: Sanity check re: value of 10GbE NICs for Hadoop?
Hadoop common provides an abstract FileSystem class, and Hadoop applications
should be designed to run on that. HDFS is just one implementation of a
valid Hadoop
adoop?
>
> Saqib
>
> -Original Message-
> From: Darren Govoni [mailto:dar...@ontrenet.com]
> Sent: Tuesday, June 28, 2011 10:21 AM
> To: common-user@hadoop.apache.org
> Subject: Re: Sanity check re: value of 10GbE NICs for Hadoop?
>
> Hadoop, like other parallel netwo
-availability or data management benefits for use
with Hadoop?
Saqib
-Original Message-
From: Darren Govoni [mailto:dar...@ontrenet.com]
Sent: Tuesday, June 28, 2011 10:21 AM
To: common-user@hadoop.apache.org
Subject: Re: Sanity check re: value of 10GbE NICs for Hadoop?
Hadoop, like other
management benefits for use
with Hadoop?
Saqib
-Original Message-
From: Darren Govoni [mailto:dar...@ontrenet.com]
Sent: Tuesday, June 28, 2011 10:21 AM
To: common-user@hadoop.apache.org
Subject: Re: Sanity check re: value of 10GbE NICs for Hadoop?
Hadoop, like other parallel networked
Hadoop, like other parallel networked computation architectures is I/O
bound, predominantly.
This means any increase in network bandwidth is "A Good Thing" and can
have drastic positive
effects on performance. All your points stem from this simple realization.
Although I'm confused by your #6.
Folks,
I've been digging into the potential benefits of using
10 Gigabit Ethernet (10GbE) NIC server connections for
Hadoop and wanted to run what I've come up with
through initial research by the list for 'sanity check'
feedback. I'd very much appreciate your input on
the importance (or lac
18 matches
Mail list logo
