Re: Parameters that can be set per job

Thanks a lot! On Fri, Jul 30, 2010 at 9:58 AM, Hemanth Yamijala wrote: > Hi, > > > Is there a list of configuration parameters that can be set per job. > > I'm almost certain there's no list that documents per-job settable > parameters that well. From 0.21 onwards, I think a convention adopted >

Re: Preferred Java version

I don't have the answer, but I know there is a _21 update available, and includes some HotSpot fixes. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Hadoop ecosystem search :: http://search-hadoop.com/ - Original Message > From: Raymond Jennings III > To: common-

Re: How to make Hadoop listen on multiple network interfaces ?

Hadoop uses DNS lookup to associate IP with hostname. It would be better if you follow the rack concept and have a ToR (Top of Rack) switch which allows for port bonding. See HBASE-2502 also. On Thu, Jul 29, 2010 at 11:57 PM, 杨杰 wrote: > Hi, everyone, > > We are now trying building a hadoop cl

How big is HDFS for CDH on EC2? :)

Hi, what I mean is, if I am running Cloudera Distribution of Hadoop on an EC2 cluster, then which areas of the physical drives of the EC2 machines are used for HDFS? Say, if I have three machines, do I get maybe 160 Gigs * 3? What if I need more space for my cluster? Thank you, Mark

Re: jobtracker.jsp reports "GC overhead limit exceeded"

Thanks, Alex. Actually I enabled all level logs. But I didn't realize to check logs in .out files and only looked at .log file and didn't see any error msgs. now I opened the .out file and saw the following logged exception: Exception in thread "IPC Server handler 5 on 50002" java.lang.OutOfMem

Re: jobtracker.jsp reports "GC overhead limit exceeded"

err, "ps aux", not "ps". Alex On Fri, Jul 30, 2010 at 3:19 PM, Alex Loddengaard wrote: > What does "ps" show you? How much memory is being used by the jobtracker, > and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)? Also > consider turning on GC logging, which will find it

Re: jobtracker.jsp reports "GC overhead limit exceeded"

What does "ps" show you? How much memory is being used by the jobtracker, and how large is its heap (loop for HADOOP_HEAPSIZE in hadoop-env.sh)? Also consider turning on GC logging, which will find its way to the jobtracker .out log in /var/log/hadoop:

jobtracker.jsp reports "GC overhead limit exceeded"

http://server:50030/jobtracker.jsp generates the following error message: HTTP ERROR: 500 GC overhead limit exceeded RequestURI=/jobtracker.jsp Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded Powered by Jetty:// The jobtracker is running below the limit. But "hadoop job -sta

How a restarted blacklisted tasktracker pick up by jobtracker?

I got a blacklisted tasktracker. After it is restarted, it is not removed from jobtracker's blacklist. The following is taken from the log of the tasktracker. ... 2010-07-30 13:08:28,475 DEBUG org.apache.hadoop.mapred.TaskTracker: Got heartbeatResponse from JobTracker with responseId: -31359 and

Re: Distance Calculation problem

Thank you Alex. I took your advice and implemented it. All the mapper calculations are being determined correctly now. Erik On 28 July 2010 14:56, Alex Kozlov wrote: > Hi Erik, > > Your assumption is that the values are coming in the same order as they > were > emitted by the mapper. This is n

Re: reuse cached files

Thanks Hemanth. Is there any way to invalidate the reuse and ask Hadoop to resent exactly the same files to cache for every job? Thanks, -Gang

Re: Preventing/Limiting NotReplicatedYetException exceptions

Thanks the information, Alex. I have mostly seen the NotReplicatedYetException issue with reduce tasks. We disabled speculative execution for reduce tasks earlier this evening so we'll see if there is an impact within the next day or so. On Mon, 26 Jul 2010 11:37 -0700, "Alex Kozlov" wrote: