Re: pointing mapred.local.dir to a ramdisk

2011-10-05 Thread Raj V
common-user@hadoop.apache.org Sent: Monday, October 3, 2011 12:31 PM Subject: Re: pointing mapred.local.dir to a ramdisk Joey Thanks. Will try and uppgrade to a newer version and check. I will also change the logs to debug and see if more information is available. Raj

Fw: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Raj V
mapred.local.dir to a ramdisk Hi all I have been trying some experiments to improve performance. One of the experiments involved pointing mapred.local.dir to a RAM disk. To this end I created a 128MB RAM disk ( each of my map outputs are smaller than this) but I have not been able to get the task tracker

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Eric Caspole
and not related to Cloudera distribution. Raj - Forwarded Message - From: Raj V rajv...@yahoo.com To: CDH Users cdh-u...@cloudera.org Sent: Friday, September 30, 2011 5:21 PM Subject: pointing mapred.local.dir to a ramdisk Hi all I have been trying some experiments to improve

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Raj V
eric.casp...@amd.com To: common-user@hadoop.apache.org Sent: Monday, October 3, 2011 8:44 AM Subject: Re: pointing mapred.local.dir to a ramdisk Are you sure you have chown'd/chmod'd the ramdisk directory to be writeable by your hadoop user? I have played with this in the past and it should basically

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Vinod Kumar Vavilapalli
@hadoop.apache.org Sent: Monday, October 3, 2011 8:44 AM Subject: Re: pointing mapred.local.dir to a ramdisk Are you sure you have chown'd/chmod'd the ramdisk directory to be writeable by your hadoop user? I have played with this in the past and it should basically work. On Oct 3, 2011

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Edward Capriolo
Caspole eric.casp...@amd.com To: common-user@hadoop.apache.org Sent: Monday, October 3, 2011 8:44 AM Subject: Re: pointing mapred.local.dir to a ramdisk Are you sure you have chown'd/chmod'd the ramdisk directory to be writeable by your hadoop user? I have played with this in the past

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Raj V
everything else the same. Raj From: Vinod Kumar Vavilapalli vino...@hortonworks.com To: common-user@hadoop.apache.org; Raj V rajv...@yahoo.com Sent: Monday, October 3, 2011 9:07 AM Subject: Re: pointing mapred.local.dir to a ramdisk Must be related to some kind

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Raj V
claim  given that I can't get it to run. -best regards Raj From: Edward Capriolo edlinuxg...@gmail.com To: common-user@hadoop.apache.org Cc: Raj V rajv...@yahoo.com Sent: Monday, October 3, 2011 10:36 AM Subject: Re: pointing mapred.local.dir to a ramdisk

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Joey Echeverria
: Monday, October 3, 2011 10:36 AM Subject: Re: pointing mapred.local.dir to a ramdisk This directory can get very large, in many cases I doubt it would fit on a ram disk. Also RAM Disks tend to help most with random read/write, since hadoop is doing mostly linear IO you may not see a great benefit from

Re: pointing mapred.local.dir to a ramdisk

2011-10-03 Thread Raj V
, October 3, 2011 11:49 AM Subject: Re: pointing mapred.local.dir to a ramdisk Raj, I just tried this on my CHD3u1 VM, and the ramdisk worked the first time. So, it's possible you've hit a bug in CDH3b3 that was later fixed. Can you enable debug logging in log4j.properties and then repost your