Re: Permissions needed to run RandomWriter ?

4096 2009-06-19 10:12 . drwxr-xr-x 5 root root 4096 2009-06-19 10:12 .. /data2/hdfs: total 8 drwxr-xr-x 2 hadoop hadoop 4096 2009-06-19 10:12 . drwxr-xr-x 4 root root 4096 2009-06-19 10:12 .. Does hadoop.tmp.dir need to be writeable by all users running hadoop jobs? -stephen -- Stephen Mu

Permissions needed to run RandomWriter ?

ermissions to /foo - do I need permissions on some other directory also or ... ? Thanks, -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research Institute, NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland http://di2.deri.iehttp://webstar.deri.iehttp://sindice.com

Re: maybe a bug in hadoop?

longer have to warn people about it. Brian On Jun 10, 2009, at 6:29 AM, stephen mulcahy wrote: Tim Wintle wrote: I thought I'd double check... $touch \ file\ with\ leading\ space $touch file\ without\ leading\ space $ ls file with leading space file without leading space Oh sure, Linux

Hadoop benchmarking

ms, each with 8 processor cores and 4 separate disks. Is there anything else I should change to stress memory more? The systems in questions have 16GB of memory but the most thats getting used during a run of this benchmark is about 2GB (and most of that seems to be os caching). Thanks, -step

Re: maybe a bug in hadoop?

ately. ... so stripping it out could mean you couldn't enter some valid directory names (but who has folders starting with a space?) Well ... I do now ;) -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research Institute, NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland http://d

maybe a bug in hadoop?

ur. I thought I should mention it just in case. Thanks, -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research Institute, NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland http://di2.deri.iehttp://webstar.deri.iehttp://sindice.com

Re: hadoop hardware configuration

case the controller mangled the disks before finally giving up the ghost. I'm very much inclined to prefer software RAID in light of this (and the fact that low-end RAID controllers have poor performance). -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research Institute, NUI Galway,

Re: Bisection bandwidth needed for a small Hadoop cluster

witches with 10 Gbps links rather than 1 Gbps links) so the ballpark number is pretty useful from a budgeting perspective. Thanks, -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research Institute, NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland http://di2.deri.iehttp://

Bisection bandwidth needed for a small Hadoop cluster

Would this be sufficient bisection bandwidth for Hadoop or should I be considering increased bandwidth between racks (maybe using fibre links between the switches or introducing another switch)? Thanks for any thoughts on this. -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research I

Re: Question about HDFS capacity and remaining

Bryan Duxbury wrote: Hm, very interesting. Didn't know about that. What's the purpose of the reservation? Just to give root preference or leave wiggle room? If it's not strictly necessary it seems like it would make sense to reduce it to essentially 0%. AFAIK It is needed for defragmentation