That was another one of our options (PXE/NFS netboot). I'm just afraid
of NFS locking on me at random times. If you've had any success with
running a live read-only root filesystem off of NFS, I'd be more
compelled in using it.
If our image is small enough (~500 MB), it may make sense to perform
netbooting, but we'd be running root over tmpfs and we would avoid NFS
after we boot the system. The same could be done if our read-only boot
device was the USB stick, and our root filesystem would run under
tmpfs. The same would apply.
On 01/30/2012 09:46 PM, Michael Segel wrote:
If you are going this route why not net boot the nodes in the cluster?
Sent from my iPhone
On Jan 30, 2012, at 8:17 PM, "Patrick Angeles"<patrickange...@gmail.com> wrote:
Hey Aaron,
I'm still skeptical when it comes to flash drives, especially as pertains
to Hadoop. The write cycle limit is impractical to make them usable for
dfs.data.dir and mapred.local.dir, and as you pointed out, you can't use
them for logs either.
If you put HADOOP_LOG_DIR in /mnt/d0, you will still have to shut down the
TT and DN in order to replace the drive. So you may as well just carve out
100GB from that drive and put your root filesystem there.
I'd say that unless you're running some extremely CPU-heavy workloads, you
should consider getting more than 3 drives per node. Most shops get 6-12
drives per node (with dual quad or hex core processors). Then you can
sacrifice one of the drives for swap and the OS.
I'd keep the RegionServer heap at 12GB or under to mitigate long GC pauses
(the bigger the heap, the longer the eventual full GC).
Finally, you can run Hive on the same cluster as HBase, just be wary of
load spikes due to MR jobs and configure properly. You don't want a large
Hive query to knock out your RegionServers thereby causing cascading
failures.
- P
On Mon, Jan 30, 2012 at 6:44 PM, Aaron Tokhy<
aaron.to...@resonatenetworks.com> wrote:
I forgot to add:
Are there use cases for using a swap partition for Hadoop nodes if our
combined planned heap size is not expected to go over 24GB for any
particular node type? I've noticed that if HBase starts to GC, it will
pause for unreasonable amounts of time if old pages get swapped to disk,
causing the regionserver to crash (which we've mitigated by setting
vm.swappiness=5).
Our slave node template will have a 1 GB heap Task Tracker, a 1 GB heap
Data Node and a 12-16GB heap RegionServer. We assume the OS memory
overhead is 1 GB. We added another 1 GB for combined Java VM overhead
across services, which comes up to be around a max of 16-20GB used. This
gives us around 4-8GB for tasks that would work with HBase. We may also
use Hive on the same cluster for queries.
On 01/30/2012 05:40 PM, Aaron Tokhy wrote:
Hi,
Our group is trying to set up a prototype for what will eventually
become a cluster of ~50 nodes.
Anyone have experiences with a stateless Hadoop cluster setup using this
method on CentOS? Are there any caveats with a read-only root file
system approach? This would save us from having to keep a root volume
on every system (whether it is installed on a USB thumb drive, or a RAID
1 of bootable / partitions).
http://citethisbook.net/Red_**Hat_Introduction_to_Stateless_**Linux.html<http://citethisbook.net/Red_Hat_Introduction_to_Stateless_Linux.html>
We would like to keep the OS root file system separate from the Hadoop
filesystem(s) for maintenance reasons (we can hot swap disks while the
system is running)
We were also considering installing the root filesystem on USB flash
drives, making it persistent yet separate. However we would identify
and turn off anything that would cause excess writes to the root
filesystem given the limited number of USB flash drive write cycles
(keep IO writes to the root filesystem to a minimum). We would do this
by storing the Hadoop logs on the same filesystem/drive as what we
specify in dfs.data.dir/dfs.name.dir.
In the end we would have something like this:
USB (MS DOS partition table + 1 ext2/3/4 partition)
/dev/sda
/dev/sda1 mounted as / (possibly read-only)
/dev/sda2 mounted as /var (read-write)
/dev/sda3 mounted as /tmp (read-write)
Hadoop Disks (no partition table or GPT since these are 3TB disks)
/dev/sdb /mnt/d0
/dev/sdc /mnt/d1
/dev/sdd /mnt/d2
/mnt/d0 would contain all Hadoop logs.
Hadoop configuration files would still reside on /
Any issues with such a setup? Are there better ways of achieving this?
