I used the default value, which I believe is 1000 MB. My cluster has about 30 machines. Each machine is configured to run up to 5 tasks. We run hourly, daily jobs on the cluster. When OOM happened, I was running a job with 1500 - 1600 mappers and 40 reducers.
I noticed that the memory usage of the job tracker keeps getting up. In one or two days, the job tracker uses about 1Gbytes memory, and stops responding to any request. Thanks. On Sat, Dec 6, 2008 at 1:45 AM, Arun C Murthy <[EMAIL PROTECTED]> wrote: > > On Dec 5, 2008, at 10:58 AM, charles du wrote: > > Any update on this? >> >> > What is the available heapsize for the JobTracker? (HADOOP_HEAPSIZE or set > it in HADOOP_JOBTRACKER_OPTS in conf/hadoop-env.sh). > > Do you remember how many total tasks (across all jobs) were executed before > the OOM? > > Arun > > > We got a similar problem after we ran a hadoop job with a lot of mappers. >> Restarting jobtracker solved the problem for a few times. But right now, >> we >> got the out of memory error right after we restarted the jobtracker. >> Thanks. >> >> >> >> >> >> On Wed, Nov 19, 2008 at 8:40 PM, Palleti, Pallavi < >> [EMAIL PROTECTED]> wrote: >> >> Hi all, >>> >>> >>> >>> We are using hadoop-0.17.2 for some time now. Since yesterday, We have >>> been seeing jobTracker failing to respond with an OutOfMemory Error very >>> frequently. Things are going fine after restarting it. But the problem >>> is occurring after a while. Below is the exception that we are seeing in >>> jobtracker logs. Can someone please suggest what is going wrong in >>> this? >>> >>> >>> >>> >>> >>> 2008-11-19 14:17:46,059 INFO org.apache.hadoop.ipc.Server: IPC Server >>> handler 9 on 9001, call >>> heartbeat([EMAIL PROTECTED], false, >>> true, 16068) >>> >>> from 205.188.170.107:51492: error: java.io.IOException: >>> java.lang.OutOfMemoryError: Java heap space >>> >>> java.io.IOException: java.lang.OutOfMemoryError: Java heap space >>> >>> at java.util.regex.Pattern.compile(Pattern.java:1452) >>> >>> at java.util.regex.Pattern.<init>(Pattern.java:1133) >>> >>> at java.util.regex.Pattern.compile(Pattern.java:847) >>> >>> at java.lang.String.replace(String.java:2208) >>> >>> at org.apache.hadoop.fs.Path.normalizePath(Path.java:146) >>> >>> at org.apache.hadoop.fs.Path.initialize(Path.java:137) >>> >>> at org.apache.hadoop.fs.Path.<init>(Path.java:126) >>> >>> at org.apache.hadoop.fs.Path.<init>(Path.java:50) >>> >>> at >>> org.apache.hadoop.mapred.Task.getTaskOutputPath(Task.java:214) >>> >>> at org.apache.hadoop.mapred.Task.setConf(Task.java:517) >>> >>> at >>> org.apache.hadoop.mapred.TaskInProgress.getTaskToRun(TaskInProgress.java >>> :745) >>> >>> at >>> org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.ja >>> va:664) >>> >>> at >>> org.apache.hadoop.mapred.JobTracker.getNewTaskForTaskTracker(JobTracker. >>> java:1585) >>> >>> at >>> org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:1309) >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav >>> a:39) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor >>> Impl.java:25) >>> >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> >>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446) >>> >>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896) >>> >>> 2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server >>> handler 4 on 9001, call >>> heartbeat([EMAIL PROTECTED], false, >>> true, 16077) f >>> >>> rom 205.188.170.84:54871: discarded for being too old (133957) >>> >>> 2008-11-19 14:18:11,869 WARN org.apache.hadoop.ipc.Server: IPC Server >>> handler 4 on 9001, call >>> heartbeat([EMAIL PROTECTED], false, >>> true, 16082) >>> >>> from 205.188.170.90:32934: discarded for being too old (133957) >>> >>> >>> >>> >>> Thanks >>> >>> Pallavi >>> >>> >>> >> >> -- >> tp >> > > -- tp
