Thanks for the suggestions. I tried the hadoop.native.lib setting (both in
job config and in hadoop-sites.xml + restart) and the problem is still
there.
I finally got the exception w/some stack trace and here it is:
2009-02-25 12:24:18,312 INFO org.apache.hadoop.mapred.TaskTracker:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:216)
at java.lang.StringBuffer.toString(StringBuffer.java:585)
at
com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1170)
at
com.sun.org.apache.xerces.internal.dom.DeferredDocumentImpl.getNodeValueString(DeferredDocumentImpl.java:1120)
at
com.sun.org.apache.xerces.internal.dom.DeferredTextImpl.synchronizeData(DeferredTextImpl.java:93)
at
com.sun.org.apache.xerces.internal.dom.CharacterDataImpl.getData(CharacterDataImpl.java:160)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:928)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:851)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:819)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:278)
at
org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:446)
at
org.apache.hadoop.mapred.JobConf.getKeepFailedTaskFiles(JobConf.java:308)
at
org.apache.hadoop.mapred.TaskTracker$TaskInProgress.setJobConf(TaskTracker.java:1497)
at
org.apache.hadoop.mapred.TaskTracker.launchTaskForJob(TaskTracker.java:727)
at
org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:721)
at
org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:1297)
at
org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:937)
at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:1334)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2343)
Tom White-3 wrote:
>
> Do you experience the problem with and without native compression? Set
> hadoop.native.lib to false to disable native compression.
>
> Cheers,
> Tom
>
> On Tue, Feb 24, 2009 at 9:40 PM, Gordon Mohr <[email protected]> wrote:
>> If you're doing a lot of gzip compression/decompression, you *might* be
>> hitting this 6+-year-old Sun JVM bug:
>>
>> "Instantiating Inflater/Deflater causes OutOfMemoryError; finalizers not
>> called promptly enough"
>> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4797189
>>
>> A workaround is listed in the issue: ensuring you call close() or end()
>> on
>> the Deflater; something similar might apply to Inflater.
>>
>> (This is one of those fun JVM situations where having more heap space may
>> make OOMEs more likely: less heap memory pressure leaves more un-GCd or
>> un-finalized heap objects around, each of which is holding a bit of
>> native
>> memory.)
>>
>> - Gordon @ IA
>>
>> bzheng wrote:
>>>
>>> I have about 24k gz files (about 550GB total) on hdfs and has a really
>>> simple
>>> java program to convert them into sequence files. If the script's
>>> setInputPaths takes a Path[] of all 24k files, it will get a OutOfMemory
>>> error at about 35% map complete. If I make the script process 2k files
>>> per
>>> job and run 12 jobs consecutively, then it goes through all files fine.
>>> The
>>> cluster I'm using has about 67 nodes. Each nodes has 16GB memory, max 7
>>> map, and max 2 reduce.
>>>
>>> The map task is really simple, it takes LongWritable as key and Text as
>>> value, generate a Text newKey, and output.collect(Text newKey, Text
>>> value). It doesn't have any code that can possibly leak memory.
>>>
>>> There's no stack trace for the vast majority of the OutOfMemory error,
>>> there's just a single line in the log like this:
>>>
>>> 2009-02-23 14:27:50,902 INFO org.apache.hadoop.mapred.TaskTracker:
>>> java.lang.OutOfMemoryError: Java heap space
>>>
>>> I can't find the stack trace right now, but rarely the OutOfMemory error
>>> originates from some hadoop config array copy opertaion. There's no
>>> special
>>> config for the script.
>>
>
>
--
View this message in context:
http://www.nabble.com/OutOfMemory-error-processing-large-amounts-of-gz-files-tp22193552p22214505.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
