In 0.20 things should get faster. Generally speaking I find HBase's insert performance really good. One of the best even. Plus Just Add Servers (tm).
-ryan On Sat, Jun 6, 2009 at 6:13 PM, llpind <sonny_h...@hotmail.com> wrote: > > Thanks Ryan, > > Yeah that sped it up a bit. > > I set : > table.setAutoFlush(false); > table.setWriteBufferSize(1024*1024*12); > > And it's inserting 1M in about 1 minute+ . Not the best still. > > 2009-06-06 18:06:54.894 ======PROCESSING RECORD: ====== @1000000 > 2009-06-06 18:08:07.725 ======PROCESSING RECORD: ====== @2000000 > 2009-06-06 18:09:24.992 ======PROCESSING RECORD: ====== @3000000 > 2009-06-06 18:11:13.279 ======PROCESSING RECORD: ====== @4000000 > > > Ryan Rawson wrote: > > > > Don't use the thrift gateway for bulk import. > > > > Use the Java API, and be sure to turn off auto flushing and use a > > reasonably > > sizable commit buffer. 1-12MB is probably ideal. > > > > i can push a 20 node cluster past 180k inserts/sec using this. > > > > On Sat, Jun 6, 2009 at 5:51 PM, llpind <sonny_h...@hotmail.com> wrote: > > > >> > >> Thanks Ryan, well done. > >> > >> I have no experience using Thrift gateway, could you please provide some > >> actual code here or in your blog post? I'd love to see how your method > >> compares with mine. > >> > >> Last night I was able to do ~58 million records in ~1.6 hours using the > >> HBase Java API directly. But with this new data, I'm seeing much slower > >> times. After reading around, it appears it's because my row key now > >> changes > >> often, whearas before it was constant for some time (more columns). > >> Thanks > >> again. :) > >> > >> > >> Ryan Rawson wrote: > >> > > >> > Have a look at: > >> > > >> > > >> > http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html > >> > > >> > -ryan > >> > > >> > > >> > On Sat, Jun 6, 2009 at 4:55 PM, llpind <sonny_h...@hotmail.com> > wrote: > >> > > >> >> > >> >> I'm doing an insert operation using the java API. > >> >> > >> >> When inserting data where the rowkey changes often, it seems the > >> inserts > >> >> go > >> >> really slow. > >> >> > >> >> Is there another method for doing inserts of this type? (instead of > >> >> BatchUpdate). > >> >> > >> >> Thanks > >> >> -- > >> >> View this message in context: > >> >> > >> > http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906724.html > >> >> Sent from the HBase User mailing list archive at Nabble.com. > >> >> > >> >> > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906943.html > >> Sent from the HBase User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23907040.html > Sent from the HBase User mailing list archive at Nabble.com. > >
