I'd be careful about adjusting HFile block size, we took 64k after benchmarking a bunch of things, and it seemed to e a good performance point.
As for scanning small rows, I'd go with a caching size of 1000-3000. When I set my scanners to that, I can pull 50k+ rows/sec from 1 client. On Thu, Nov 11, 2010 at 7:36 AM, Friso van Vollenhoven <fvanvollenho...@xebia.com> wrote: >> Great , thank you for the explanation. >> >> my table schema is: >> >> {NAME => 'URLs_sanity', FAMILIES => [{NAME => 'gs', VERSIONS => >> '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', >> IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'meta-data', VERSIONS >> => '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', >> IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'snt', VERSIONS => >> '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536', >> IN_MEMORY => 'false', BLOCKCACHE => 'true'}] >> >> couple of questions: >> 1) How can I know what is the optimal size of BlockSize? What is the >> best practice regarding this issue > > Check the link I sent. There is an explanation on this setting in there. > >> 2) Assuming that I have a record 4 k and changed to 50 --> 4*50 = 200 >> and it is ~ 3 blocks , so performance had to be improved , but execution >> time was the same. > > There is of course more involved than just this. And also, you may be already > getting the most of what your hardware can give you. You should also try to > find out what bottleneck you have (IO or CPU or network). Hadoop and HBase > have many settings. There is no magic single knob that makes things fast or > slow. > >> >> Oleg. >> >> >> On Thu, Nov 11, 2010 at 3:08 PM, Friso van Vollenhoven < >> fvanvollenho...@xebia.com> wrote: >> >>> Not that block size (that's the HDFS one), but the HBase block size. You >>> set it at table creation or it uses the default of 64K. >>> >>> The description of hbase.client.scanner.caching says: >>> Number of rows that will be fetched when calling next >>> on a scanner if it is not served from memory. Higher caching values >>> will enable faster scanners but will eat up more memory and some >>> calls of next may take longer and longer times when the cache is empty. >>> >>> That means that it will pre-fetch that number of rows, if the next row does >>> not come from memory. So if your rows are small enough to fit 100 of them in >>> one block, it doesn't matter whether you pre-fetch 1, 50 or 99, because it >>> will only go to disk when it exhausts the whole block, which sticks in block >>> cache. So, it will still fetch the same amount of data from disk every time. >>> If you increase the number to a value that is certain to load multiple >>> blocks at a time from disk, it will increase performance. >>> >>> >>> >>> On 11 nov 2010, at 12:55, Oleg Ruchovets wrote: >>> >>>> Yes , I thought about large number , so you said it depends on block >>> size. >>>> Good point. >>>> >>>> I have one recored ~ 4k , >>>> block size is: >>>> >>>> <property> >>>> <name>dfs.block.size</name> >>>> <value>268435456</value> >>>> <description>HDFS blocksize of 256MB for large file-systems. >>>> </description> >>>> </property> >>>> >>>> what is the number that I have choose? Assuming >>>> I am afraid that using number which is equal one block brings to >>>> socketTimeOutException? Am I write? >>>> >>>> Thanks Oleg. >>>> >>>> >>>> >>>> >>>> On Thu, Nov 11, 2010 at 1:30 PM, Friso van Vollenhoven < >>>> fvanvollenho...@xebia.com> wrote: >>>> >>>>> How small is small? If it is bytes, then setting the value to 50 is not >>> so >>>>> much different from 1, I suppose. If 50 rows fit in one block, it will >>> just >>>>> fetch one block whether the setting is 1 or 50. You might want to try a >>>>> larger value. It should be fine if the records are small and you need >>> them >>>>> all on the client side anyway. >>>>> >>>>> It also depends on the block size, of course. When you only ever do full >>>>> scans on a table and little random access, you might want to increase >>> that. >>>>> >>>>> Friso >>>>> >>>>> >>>>> >>>>> >>>>> On 11 nov 2010, at 12:15, Oleg Ruchovets wrote: >>>>> >>>>>> Hi , >>>>>> To improve client performance I changed >>>>>> hbase.client.scanner.caching from 1 to 50. >>>>>> After running client with new value( hbase.client.scanner.caching from >>> = >>>>> 50 >>>>>> ) it didn't improve execution time at all. >>>>>> >>>>>> I have ~ 9 million small records. >>>>>> I have to do full scan , so it brings all 9 million records to client >>> . >>>>>> My assumption -- this change have to bring significant improvement , >>> but >>>>> it >>>>>> is not. >>>>>> >>>>>> Additional Information. >>>>>> I scan table which has 100 regions >>>>>> 5 server >>>>>> 20 map >>>>>> 4 concurrent map >>>>>> Scan process takes 5.5 - 6 hours. As for me it is too much time? Am I >>>>> write? >>>>>> and how can I improve it >>>>>> >>>>>> >>>>>> I changed the value in all hbase-site.xml files and restart hbase. >>>>>> >>>>>> Any suggestions. >>>>> >>>>> >>> >>> > >