Attached is a simple code I wrote for loading data into a HBase table with MR. Hope this helps. Also I have two questions,
1. It runs very well. However, when a reducer arrives 66% complete, or in other words, when copy/sort is done, the percentage complete stays at 66% until some reducer is 100% completed. There is no such problem if I use old Hadoop/HBase 0.19 API. Is it a HBase 0.20.0 bug? Or some configuration issues? 2. By default, a reducer fails if it does not update its status to namenode in 10 minutes. So to avoid failure when loading huge data, at line 91 it reports status for every 10000 records loaded. It works great to address the failure issue. However, I also wanted to report status after all records are loaded and return the total number of records being loaded at line 97. Very weirdly, the info printed at line 97 always overlaps the info printed at line 91, so I never see the info printed at line 91 is displayed on Web UI, during the executaion of the reducer. Can sb take a look? terryg wrote: > > Thank you. I'm playing around with that example now. > > stack-3 wrote: >> >> From the same package, there is IdentityTableReduce for the most basic >> reduce and then IndexTableReducer (See BuildTableIndex for how to hook it >> up). >> St.Ack >> >> On Sat, Sep 26, 2009 at 6:50 AM, terryg <fontspec-webs...@yahoo.com> >> wrote: >> >>> >>> Thank you for your reply. I had previously found Kevin Peterson's >>> example, >>> but it lacks a reduce step as does the RowCount example. Does anyone >>> know >>> of a good 0.20 reduce example? >>> >>> stack-3 wrote: >>> > >>> > Or better, >>> > >>> http://kdpeterson.net/blog/2009/09/minimal-hbase-mapreduce-example.html >>> > St.Ack >>> > >>> > On Fri, Sep 25, 2009 at 2:32 PM, stack <st...@duboce.net> wrote: >>> > >>> >> On Fri, Sep 25, 2009 at 2:09 PM, terryg >>> >> <fontspec-webs...@yahoo.com>wrote: >>> >> >>> >>> >>> >>> I'm working on a map/reduce java program for performing a comparison >>> of >>> >>> columns in a single record and then reporting the percent matched. >>> >>> From >>> >>> what I have read the comparison portion would occur during the map >>> step, >>> >>> and >>> >>> the reduce step would then aggregate the totals to come up with a >>> >>> percent >>> >>> matched for each field. >>> >>> >>> >>> What i'm trying to find are some simple map/reduce hbase examples >>> for >>> >>> 0.20 >>> >>> API. Can anyone point me in the direction of some? >>> >> >>> >> >>> >> See content of org.apache.hadoop.hbase.mapreduce package. Checkout >>> in >>> >> particular the RowCounter. >>> >> Yours, >>> >> St.Ack >>> >> >>> >> >>> > >>> > >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Looking-for-simple-hbase-map-reduce-example-tp25619336p25625432.html >>> Sent from the HBase User mailing list archive at Nabble.com. >>> >>> >> >> > > http://www.nabble.com/file/p25651120/BulkLoader.java BulkLoader.java http://www.nabble.com/file/p25651120/BulkLoader.java BulkLoader.java -- View this message in context: http://www.nabble.com/Looking-for-simple-hbase-map-reduce-example-tp25619336p25651120.html Sent from the HBase User mailing list archive at Nabble.com.
