I opened https://issues.apache.org/jira/browse/HBASE-2063 to cover this issue. Thanks, St.Ack
On Mon, Dec 21, 2009 at 8:35 AM, stack <[email protected]> wrote: > I was going to write that perhaps you needed to > turn mapred.reduce.tasks.speculative.execution off, but if enabling it and > things work, that would seem to indicate that a our reducer first takes > longer than the task timeout maximum and secondly, on failure, we should > clean up the hfile. > > On the first issue, you are using KeyValueSortReducer? Are your values > large? We set reducer status every 100 values. Maybe this is not enough? > We should set status more frequently? If you call context setstatus more > frequently, do things work w/o speculative execution? > > On the second, HFileOutputFormat close will set the metadata on the hfile > and then close it. On kill, this code is not being called. Let me see if > can do something about that (e.g. register a shutdown hook to clean away > incomplete files -- ). > > Thanks, > St.Ack > > > On Sun, Dec 20, 2009 at 11:26 PM, ChingShen <[email protected]>wrote: > >> I think I found a way. >> I set the "mapred.reduce.tasks.speculative.execution" to true and output >> hfiles again, then successfully load hfiles into hbase. >> Is it best solution? or HFileOutputFormat bug? >> >> Shen >> >> On Mon, Dec 21, 2009 at 8:25 AM, ChingShen <[email protected]> >> wrote: >> >> > Thanks, stack. >> > >> > I checked this file that isn't empty. But I found that as long as the >> > "Killed Task Attempts" > 0 in reduce phase, and run the loadtable.rb >> script >> > to load hfiles then failed. >> > How to avoid this problem? >> > >> > Thanks. >> > >> > Shen >> > >> > >> > On Sat, Dec 19, 2009 at 3:49 AM, stack <[email protected]> wrote: >> > >> >> Check the >> >> file >> >> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/197894389945760574. >> >> Is it empty? Was there an error during running of your MR job? >> Perhaps >> >> a >> >> task failed? >> >> >> >> St.Ack >> >> >> >> >> >> >> >> On Thu, Dec 17, 2009 at 9:46 PM, ChingShen <[email protected]> >> >> wrote: >> >> >> >> > Hi, >> >> > I use the script loadtable.rb to load my hfiles into hbase, but I >> got >> >> an >> >> > exception as below. >> >> > Does anyone have any suggestions? >> >> > >> >> > 09/12/17 23:59:33 INFO loadtable: 18 read firstkey of -3.9290_52.5534 >> >> from >> >> > >> >> > >> >> >> hdfs://domU-12-31-39-09-C5-54.compute-1.internal/osm2_hfile/Level4/1978943899457605747 >> >> > org/apache/hadoop/hbase/io/hfile/HFile.java:1335:in `deserialize': >> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size >> >> match >> >> > content? (NativeException) >> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:813:in >> `readTrailer' >> >> > from org/apache/hadoop/hbase/io/hfile/HFile.java:758:in >> >> `loadFileInfo' >> >> > from sun.reflect.GeneratedMethodAccessor7:-1:in `invoke' >> >> > from sun/reflect/DelegatingMethodAccessorImpl.java:25:in `invoke' >> >> > from java/lang/reflect/Method.java:597:in `invoke' >> >> > from org/jruby/javasupport/JavaMethod.java:298:in >> >> > `invokeWithExceptionHandling' >> >> > from org/jruby/javasupport/JavaMethod.java:259:in `invoke' >> >> > from org/jruby/java/invokers/InstanceMethodInvoker.java:36:in >> `call' >> >> > ... 18 levels... >> >> > from org/jruby/Main.java:94:in `main' >> >> > from loadtable.rb:83:in `each' >> >> > from loadtable.rb:83 >> >> > Complete Java stackTrace >> >> > java.io.IOException: Trailer 'header' is wrong; does the trailer size >> >> match >> >> > content? >> >> > at >> >> > >> >> > >> >> >> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1335) >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:813) >> >> > at >> >> > >> >> >> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:758) >> >> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) >> >> > at >> >> > >> >> > >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> > at java.lang.reflect.Method.invoke(Method.java:597) >> >> > at >> >> > >> >> > >> >> >> org.jruby.javasupport.JavaMethod.invokeWithExceptionHandling(JavaMethod.java:298) >> >> > at org.jruby.javasupport.JavaMethod.invoke(JavaMethod.java:259) >> >> > at >> >> > >> >> > >> >> >> org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) >> >> > at >> >> > >> org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:70) >> >> > at loadtable.ensure_1$RUBY$__ensure___2(loadtable.rb:86) >> >> > at loadtable.block_0$RUBY$__for__(loadtable.rb:85) >> >> > at loadtableBlockCallback$block_0$RUBY$__for__xx1.call(Unknown >> >> Source) >> >> > at org.jruby.runtime.CompiledBlock.yield(CompiledBlock.java:102) >> >> > at org.jruby.runtime.Block.yield(Block.java:100) >> >> > at >> >> org.jruby.java.proxies.ArrayJavaProxy.each(ArrayJavaProxy.java:112) >> >> > at >> >> > >> >> > >> >> >> org.jruby.java.proxies.ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.call(org/jruby/java/proxies/ArrayJavaProxy$i_method_0_0$RUBYINVOKER$each.gen) >> >> > at >> >> > >> >> > >> >> >> org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:263) >> >> > at >> >> > >> >> > >> >> >> org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:81) >> >> > at >> >> > >> >> > >> >> >> org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:96) >> >> > at loadtable.__file__(loadtable.rb:83) >> >> > at loadtable.load(loadtable.rb) >> >> > at org.jruby.Ruby.runScript(Ruby.java:577) >> >> > at org.jruby.Ruby.runNormally(Ruby.java:480) >> >> > at org.jruby.Ruby.runFromMain(Ruby.java:354) >> >> > at org.jruby.Main.run(Main.java:229) >> >> > at org.jruby.Main.run(Main.java:110) >> >> > at org.jruby.Main.main(Main.java:94) >> >> > >> >> >> > >> > >> > >> > >> >> >> -- >> ***************************************************** >> Ching-Shen Chen >> Advanced Technology Center, >> Information & Communications Research Lab. >> E-mail: [email protected] >> Tel:+886-3-5915542 >> ***************************************************** >> > >
