On 05/10/2011 06:56 AM, Jonathan Disher wrote:
In a previous life, I've had extreme problems with XFS, including
kernel panics and data loss under high load.
Those were database servers, not Hadoop nodes, and it was a few years
ago. But, ext3/ext4 seems to be stable enough, and it's more widely
supported, so it's my preference.
-j
On May 10, 2011, at 3:59 AM, Rita wrote:
I keep asking because I wasn't able to use a XFS filesystem larger
than 3-4TB. If the XFS file system is larger than 4TB hdfs won't
recognize the space. I am on a 64bit RHEL 5.3 host.
On Tue, May 10, 2011 at 6:30 AM, Will Maier <wcma...@hep.wisc.edu
<mailto:wcma...@hep.wisc.edu>> wrote:
On Tue, May 10, 2011 at 12:03:09AM -0400, Rita wrote:
> what filesystem are they using and what is the size of each
filesystem?
It sounds nuts, but each disk has its own ext3 filesystem. Beyond
switching to
the deadline IO scheduler, we haven't done much tuning/tweaking.
A script runs
every ten minutes to test all of the data mounts and reconfigure
hdfs-site.xml
and restart the datanode if necessary. So far, this approach has
allowed us to
avoid loss of space to RAID without correlating the risk of disk
failure by
building larger RAID0s.
In the future, we expect to deprecate the script and rely on the
datanode process
itself to handle missing/failing disks.
--
Will Maier - UW High Energy Physics
cel: 608.438.6162 <tel:608.438.6162>
tel: 608.263.9692 <tel:608.263.9692>
web: http://www.hep.wisc.edu/~wcmaier/
<http://www.hep.wisc.edu/%7Ewcmaier/>
--
--- Get your facts first, then you can distort them as you please.--
Jonathan, I had the same issues on my PostgreSQL servers, and the main
issues was given by the kernel version that I was using.
I upgrade the kernel to the last version supported by Red Hat, and
everything worked OK.
My prefered filesystem is ZFS, It's a shame that Linux support is very
inmature yet. For that reason, I changed my PostgreSQL hosts to
FreeBSD-8.0 to use
ZFS like filesystem and it's really rocks.
Had anyone tested a Hadoop cluster with this filesystem?
On Solaris or FreeBSD?
Regards
--
Marcos Luís Ortíz Valmaseda
Software Engineer (Large-Scaled Distributed Systems)
University of Information Sciences,
La Habana, Cuba
Linux User # 418229
http://about.me/marcosortiz
