We could create an apache hadoop list of product selection discussions. I
believe this list is intended to be focused on project governance and similar
discussions. Maybe we should simply create a governance list and leave this
one to be the free for all?
On Jul 2, 2011, at 9:16 PM, Ian
On Wed, Jul 13, 2011 at 7:59 AM, Eric Baldeschwieler eri...@hortonworks.com
wrote:
We could create an apache hadoop list of product selection discussions. I
believe this list is intended to be focused on project governance and
similar discussions. Maybe we should simply create a governance
On Jul 2, 2011, at 7:38 AM, Ted Dunning wrote:
On Fri, Jul 1, 2011 at 11:53 AM, Abhishek Mehta abhis...@tresata.comwrote:
open and honest conversations on this thread/group, irrespective of
whether one represents a company or apache, should be encouraged.
Paradoxically, I partially
Keeping the amount of disks per node low and the amount of nodes high
should keep the impact of dead nodes in control.
It keeps the impact of dead nodes in control but I don't think thats
long-term cost efficient. As prices of 10GbE go down, the keep the node
small arguement seems less
What's the justification for a management interface? Doesn't that increase
complexity? Also you still twice the ports?
On Jun 30, 2011 11:54 PM, Evert Lammerts evert.lamme...@sara.nl wrote:
Keeping the amount of disks per node low and the amount of nodes high
should keep the impact of dead nodes
What's the justification for a management interface? Doesn't that increase
complexity? Also you still twice the ports?
That's true. It's a tradition that I haven't questioned before. The reasoning,
whether right or wrong, is that user jobs (our users are external) can get in
the way of
On 01/07/2011 07:41, Evert Lammerts wrote:
Keeping the amount of disks per node low and the amount of nodes high
should keep the impact of dead nodes in control.
It keeps the impact of dead nodes in control but I don't think thats
long-term cost efficient. As prices of 10GbE go down, the keep
On 01/07/2011 08:16, Ryan Rawson wrote:
What's the justification for a management interface? Doesn't that increase
complexity? Also you still twice the ports?
ILO reduces ops complexity. you can push things like BIOS updates out,
boot machines into known states, instead of the slowly
On 01/07/2011 01:16, Ted Dunning wrote:
You have to consider the long-term reliability as well.
Losing an entire set of 10 or 12 disks at once makes the overall reliability
of a large cluster very suspect. This is because it becomes entirely too
likely that two additional drives will fail
You are twice the ports, but if you know that the management port is not
used for serious data, then you can put a SOHO grade switch on those ports
at negligible cost.
There is a serious conflict of goals here if you have software that can make
serious use more than one NIC. On the one hand, it
Although this thread is wandering a bit, I disagree strongly that it is
inappropriate to discuss other vendor specific features (or competing
compute platform features) on general@. The topic has become the factors
that influence hardware purchase choices, and one of those is how the
system deals
On Fri, Jul 1, 2011 at 9:12 AM, Steve Loughran ste...@apache.org wrote:
On 01/07/2011 01:16, Ted Dunning wrote:
You have to consider the long-term reliability as well.
Losing an entire set of 10 or 12 disks at once makes the overall
reliability
of a large cluster very suspect. This is
i definitely agree with scott. as a user of the hadoop open source stack for
building our banking focused big data analytics applications, i speak on behalf
of our clients and the emerging hadoop eco-system that open and honest
conversations on this thread/group, irrespective of whether one
On Fri, Jul 1, 2011 at 11:53 AM, Abhishek Mehta abhis...@tresata.comwrote:
open and honest conversations on this thread/group, irrespective of
whether one represents a company or apache, should be encouraged.
Paradoxically, I partially agree with Ian on this. On the one hand, it is
You can get 12-24 TB in a server today, which means the loss of a server
generates a lot of traffic -which argues for 10 Gbe.
But
-big increase in switch cost, especially if you (CoI warning) go with
Cisco
-there have been problems with things like BIOS PXE and lights out
management
11:38 PM
To: Evert Lammerts
Subject: Fwd: Hadoop Java Versions
what are the other switch options (other than cisco that is?)?
cheers
Abhishek Mehta
(e) abhis...@tresata.commailto:abhis...@tresata.com
(v) 980.355.9855
Begin forwarded message:
From: Evert Lammerts evert.lamme
Keeping the amount of disks per node low and the amount of nodes high
should keep the impact of dead nodes in control.
It keeps the impact of dead nodes in control but I don't think thats
long-term cost efficient. As prices of 10GbE go down, the keep the node
small arguement seems less fitting.
You have to consider the long-term reliability as well.
Losing an entire set of 10 or 12 disks at once makes the overall reliability
of a large cluster very suspect. This is because it becomes entirely too
likely that two additional drives will fail before the data on the off-line
node can be
On Thu, Jun 30, 2011 at 5:16 PM, Ted Dunning tdunn...@maprtech.com wrote:
You have to consider the long-term reliability as well.
Losing an entire set of 10 or 12 disks at once makes the overall
reliability
of a large cluster very suspect. This is because it becomes entirely too
likely
Good point Todd.
I was speaking from the experience of people I know who are using 0.20.x
On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon t...@cloudera.com wrote:
On Thu, Jun 30, 2011 at 5:16 PM, Ted Dunning tdunn...@maprtech.com
wrote:
You have to consider the long-term reliability as well.
On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon t...@cloudera.com wrote:
I'd advise you to look at stock hadoop again. This used to be true, but
was fixed a long while back by HDFS-457 and several followup JIRAs.
If MapR does something fancier, I'm sure we'd be interested to hear about
it
so
On Jul 1, 2011, at 2:08 PM, M. C. Srivas wrote:
On Thu, Jun 30, 2011 at 5:24 PM, Todd Lipcon t...@cloudera.com wrote:
I'd advise you to look at stock hadoop again. This used to be true, but
was fixed a long while back by HDFS-457 and several followup JIRAs.
If MapR does something
No worries.
I read Todd's post as asking for elaboration ... sometimes knowing what
another similar system does helps in improving your own.
On Thu, Jun 30, 2011 at 9:47 PM, Ian Holsman had...@holsman.net wrote:
On Jul 1, 2011, at 2:08 PM, M. C. Srivas wrote:
On Thu, Jun 30, 2011 at 5:24
Ian,
Good point. Srivas was responding to Todd's question, but there might be
better fora as you suggest.
We have a good one for specific questions about MapR at
http://answers.mapr.com
That doesn't, however, really provide a useful forum for questions like
Todd's which really spans both
On 28/06/11 04:49, Segel, Mike wrote:
Hmmm. I could have sworn there was a background balancing bandwidth limiter.
There is, for the rebalancer, node outages are taken more seriously,
though there have been problems in past 0.20.x where there was a risk of
a cascade failure on a big
You're preaching to the choir. :-)
With Sandybridge, you're going to start seeing 10 GBe on the motherboard.
We built our clusters using 1U boxes where you're stuck w 4 3.5 drives. With
larger chassis,
You can fit an additional controller card and more drives.
More drives reduces the bottleneck
We at Yahoo are about to deploy code to ensure a disk failure on a datanode is
just that - a disk failure. Not a node failure. This really helps avoid
replication storms.
It's in the 0.20.204 branch for the curious.
Arun
Sent from my iPhone
On Jun 28, 2011, at 3:01 AM, Steve Loughran
On 26/06/11 20:23, Scott Carey wrote:
On 6/23/11 5:49 AM, Steve Loughranste...@apache.org wrote:
what's your HW setup? #cores/server, #servers, underlying OS?
CentOS 5.6.
4 cores / 8 threads a server (Nehalem generation Intel processor).
that should be enough to find problems. I've
On the subject of gige vs 10-gige, I think that we will very shortly
be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
drive of streaming data. Nodes with 4+ disks are throttled by the
network. On a small cluster (20 nodes), the replication traffic can
choke a cluster to death.
Come to Srivas talk at the Summit.
On Mon, Jun 27, 2011 at 5:10 PM, Ryan Rawson ryano...@gmail.com wrote:
On the subject of gige vs 10-gige, I think that we will very shortly
be seeing interest in 10gig, since gige is only 120MB/sec - 1 hard
drive of streaming data. Nodes with 4+ disks are
That doesn't seem right.
In one of our test clusters (19 data nodes) we found that under heavy loads we
were disk I/O bound and not network bound. Of course YMMV depending on your ToR
switch. If we had more than 4 disks per node, we would probably see the network
being the bottleneck. What did
There are no bandwidth limitations in 0.20.x. None that I saw at
least. It was basically bandwidth-management-by-pwm. You could
adjust the frequency of how many files-per-node were copied.
In my case, the load was HBase real time serving, so it was servicing
more smaller random reads, not a
For cost reasons, we just bonded two 1G network ports together. A single
stream won't go past 1Gbps, but concurrent ones do -- this is with the
Linux built-in bonding. The network is only stressed during 'sort-like'
jobs or big replication events.
We also removed some disk bottlenecks by tuning
Hmmm. I could have sworn there was a background balancing bandwidth limiter.
Haven't tested random reads... The last test we did ended up hitting the cache,
but we didn't push it hard enough to hit network bandwidth limitations... Not
to say they don't exist. Like I said in the other post, if
On 6/23/11 5:49 AM, Steve Loughran ste...@apache.org wrote:
On 22/06/2011 21:27, Scott Carey wrote:
Problems have been reported with Hadoop, the 64-bit JVM and Compressed
Object References (the -XX:+UseCompressedOops option), so use of that
option is discouraged.
I think the above is
On 22/06/2011 21:27, Scott Carey wrote:
Problems have been reported with Hadoop, the 64-bit JVM and Compressed
Object References (the -XX:+UseCompressedOops option), so use of that
option is discouraged.
I think the above is dated. It also lacks critical information. What JVM
and OS version
Problems have been reported with Hadoop, the 64-bit JVM and Compressed
Object References (the -XX:+UseCompressedOops option), so use of that
option is discouraged.
I think the above is dated. It also lacks critical information. What JVM
and OS version was the problem seen?
CompressedOops had
On Jun 22, 2011, at 1:27 PM, Scott Carey wrote:
Problems have been reported with Hadoop, the 64-bit JVM and Compressed
Object References (the -XX:+UseCompressedOops option), so use of that
option is discouraged.
I think the above is dated. It also lacks critical information. What JVM
and
On 6/22/11 1:49 PM, Allen Wittenauer a...@apache.org wrote:
On Jun 22, 2011, at 1:27 PM, Scott Carey wrote:
Problems have been reported with Hadoop, the 64-bit JVM and Compressed
Object References (the -XX:+UseCompressedOops option), so use of that
option is discouraged.
I think the
39 matches
Mail list logo
