I agree seams in 0.20 that I am testing my clients are the bottleneck now not the db I went from seeing 30K on 8 nodes to around 115k when I could keep all the clients writing at the same time for a few secs.

Billy



"Ryan Rawson" <[email protected]> wrote in message news:[email protected]...
In 0.20 things should get faster.

Generally speaking I find HBase's insert performance really good.  One of
the best even.  Plus Just Add Servers (tm).

-ryan

On Sat, Jun 6, 2009 at 6:13 PM, llpind <[email protected]> wrote:


Thanks Ryan,

Yeah that sped it up a bit.

I set :
               table.setAutoFlush(false);
               table.setWriteBufferSize(1024*1024*12);

And it's inserting 1M in about 1 minute+ .   Not the best still.

2009-06-06 18:06:54.894 ======PROCESSING RECORD: ====== @1000000
2009-06-06 18:08:07.725 ======PROCESSING RECORD: ====== @2000000
2009-06-06 18:09:24.992 ======PROCESSING RECORD: ====== @3000000
2009-06-06 18:11:13.279 ======PROCESSING RECORD: ====== @4000000


Ryan Rawson wrote:
>
> Don't use the thrift gateway for bulk import.
>
> Use the Java API, and be sure to turn off auto flushing and use a
> reasonably
> sizable commit buffer. 1-12MB is probably ideal.
>
> i can push a 20 node cluster past 180k inserts/sec using this.
>
> On Sat, Jun 6, 2009 at 5:51 PM, llpind > <[email protected]> wrote:
>
>>
>> Thanks Ryan, well done.
>>
>> I have no experience using Thrift gateway, could you please provide >> some >> actual code here or in your blog post? I'd love to see how your >> method
>> compares with mine.
>>
>> Last night I was able to do ~58 million records in ~1.6 hours using >> the >> HBase Java API directly. But with this new data, I'm seeing much >> slower
>> times.  After reading around, it appears it's because my row key now
>> changes
>> often, whearas before it was constant for some time (more columns).
>> Thanks
>> again. :)
>>
>>
>> Ryan Rawson wrote:
>> >
>> > Have a look at:
>> >
>> >
>>
http://ryantwopointoh.blogspot.com/2009/01/performance-of-hbase-importing.html
>> >
>> > -ryan
>> >
>> >
>> > On Sat, Jun 6, 2009 at 4:55 PM, llpind >> > <[email protected]>
wrote:
>> >
>> >>
>> >> I'm doing an insert operation using the java API.
>> >>
>> >> When inserting data where the rowkey changes often, it seems the
>> inserts
>> >> go
>> >> really slow.
>> >>
>> >> Is there another method for doing inserts of this type? (instead >> >> of
>> >> BatchUpdate).
>> >>
>> >> Thanks
>> >> --
>> >> View this message in context:
>> >>
>>
http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906724.html
>> >> Sent from the HBase User mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> >
>>
>> --
>> View this message in context:
>>
http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23906943.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
>

--
View this message in context:
http://www.nabble.com/Frequent-changing-rowkey---HBase-insert-tp23906724p23907040.html
Sent from the HBase User mailing list archive at Nabble.com.





Reply via email to