Silly question…
Why does the idea of using versioning to capture temporal changes to data keep
being propagated?
Seriously this issue keeps popping up…
If you want to capture data over time… use a timestamp as part of the column
name. Don’t abuse the cell’s version.
On Apr 11, 2014, at 11:03 AM, gortiz <gor...@pragsis.com> wrote:
> Yes, I have tried with two different values for that value of versions, 1000
> and maximum value for integers.
>
> But, I want to keep those versions. I don't want to keep just 3 versions.
> Imagine that I want to record a new version each minute and store a day,
> those are 1440 versions.
>
> Why is HBase going to read all the versions?? , I thought, if you don't
> indicate any versions it's just read the newest and skip the rest. It doesn't
> make too much sense to read all of them if data is sorted, plus the newest
> version is stored in the top.
>
>
> On 11/04/14 11:54, Anoop John wrote:
>> What is the max version setting u have done for ur table cf? When u set
>> some a value, HBase has to keep all those versions. During a scan it will
>> read all those versions. In 94 version the default value for the max
>> versions is 3. I guess you have set some bigger value. If u have not,
>> mind testing after a major compaction?
>>
>> -Anoop-
>>
>> On Fri, Apr 11, 2014 at 1:01 PM, gortiz <gor...@pragsis.com> wrote:
>>
>>> Last test I have done it's to reduce the number of versions to 100.
>>> So, right now, I have 100 rows with 100 versions each one.
>>> Times are: (I got the same times for blocksize of 64Ks and 1Mb)
>>> 100row-1000versions + blockcache-> 80s.
>>> 100row-1000versions + No blockcache-> 70s.
>>>
>>> 100row-*100*versions + blockcache-> 7.3s.
>>> 100row-*100*versions + No blockcache-> 6.1s.
>>>
>>> What's the reasons of this? I guess HBase is enough smart for not consider
>>> old versions, so, it just checks the newest. But, I reduce 10 times the
>>> size (in versions) and I got a 10x of performance.
>>>
>>> The filter is scan 'filters', {FILTER => "ValueFilter(=,
>>> 'binary:5')",STARTROW => '1010000000000000000000000000000000000101',
>>> STOPROW => '6010000000000000000000000000000000000201'}
>>>
>>>
>>>
>>> On 11/04/14 09:04, gortiz wrote:
>>>
>>>> Well, I guessed that, what it doesn't make too much sense because it's so
>>>> slow. I only have right now 100 rows with 1000 versions each row.
>>>> I have checked the size of the dataset and each row is about 700Kbytes
>>>> (around 7Gb, 100rowsx1000versions). So, it should only check 100 rows x
>>>> 700Kbytes = 70Mb, since it just check the newest version. How can it spend
>>>> too many time checking this quantity of data?
>>>>
>>>> I'm generating again the dataset with a bigger blocksize (previously was
>>>> 64Kb, now, it's going to be 1Mb). I could try tunning the scanning and
>>>> baching parameters, but I don't think they're going to affect too much.
>>>>
>>>> Another test I want to do, it's generate the same dataset with just
>>>> 100versions, It should spend around the same time, right? Or am I wrong?
>>>>
>>>> On 10/04/14 18:08, Ted Yu wrote:
>>>>
>>>>> It should be newest version of each value.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> On Thu, Apr 10, 2014 at 9:55 AM, gortiz <gor...@pragsis.com> wrote:
>>>>>
>>>>> Another little question is, when the filter I'm using, Do I check all the
>>>>>> versions? or just the newest? Because, I'm wondering if when I do a scan
>>>>>> over all the table, I look for the value "5" in all the dataset or I'm
>>>>>> just
>>>>>> looking for in one newest version of each value.
>>>>>>
>>>>>>
>>>>>> On 10/04/14 16:52, gortiz wrote:
>>>>>>
>>>>>> I was trying to check the behaviour of HBase. The cluster is a group of
>>>>>>> old computers, one master, five slaves, each one with 2Gb, so, 12gb in
>>>>>>> total.
>>>>>>> The table has a column family with 1000 columns and each column with
>>>>>>> 100
>>>>>>> versions.
>>>>>>> There's another column faimily with four columns an one image of 100kb.
>>>>>>> (I've tried without this column family as well.)
>>>>>>> The table is partitioned manually in all the slaves, so data are
>>>>>>> balanced
>>>>>>> in the cluster.
>>>>>>>
>>>>>>> I'm executing this sentence *scan 'table1', {FILTER => "ValueFilter(=,
>>>>>>> 'binary:5')"* in HBase 0.94.6
>>>>>>> My time for lease and rpc is three minutes.
>>>>>>> Since, it's a full scan of the table, I have been playing with the
>>>>>>> BLOCKCACHE as well (just disable and enable, not about the size of
>>>>>>> it). I
>>>>>>> thought that it was going to have too much calls to the GC. I'm not
>>>>>>> sure
>>>>>>> about this point.
>>>>>>>
>>>>>>> I know that it's not the best way to use HBase, it's just a test. I
>>>>>>> think
>>>>>>> that it's not working because the hardware isn't enough, although, I
>>>>>>> would
>>>>>>> like to try some kind of tunning to improve it.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 10/04/14 14:21, Ted Yu wrote:
>>>>>>>
>>>>>>> Can you give us a bit more information:
>>>>>>>> HBase release you're running
>>>>>>>> What filters are used for the scan
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Apr 10, 2014, at 2:36 AM, gortiz <gor...@pragsis.com> wrote:
>>>>>>>>
>>>>>>>> I got this error when I execute a full scan with filters about a
>>>>>>>> table.
>>>>>>>>
>>>>>>>>> Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.
>>>>>>>>> regionserver.LeaseException:
>>>>>>>>> org.apache.hadoop.hbase.regionserver.LeaseException: lease
>>>>>>>>> '-4165751462641113359' does not exist
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.hbase.regionserver.Leases.removeLease(Leases.java:231)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> at org.apache.hadoop.hbase.regionserver.HRegionServer.
>>>>>>>>> next(HRegionServer.java:2482)
>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>>>> NativeMethodAccessorImpl.java:39)
>>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>>>> DelegatingMethodAccessorImpl.java:25)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(
>>>>>>>>> WritableRpcEngine.java:320)
>>>>>>>>> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(
>>>>>>>>> HBaseServer.java:1428)
>>>>>>>>>
>>>>>>>>> I have read about increase the lease time and rpc time, but it's not
>>>>>>>>> working.. what else could I try?? The table isn't too big. I have
>>>>>>>>> been
>>>>>>>>> checking the logs from GC, HMaster and some RegionServers and I
>>>>>>>>> didn't see
>>>>>>>>> anything weird. I tried as well to try with a couple of caching
>>>>>>>>> values.
>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>>> *Guillermo Ortiz*
>>>>>> /Big Data Developer/
>>>>>>
>>>>>> Telf.: +34 917 680
>>>>>> 490<https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#>
>>>>>> Fax: +34 913 833
>>>>>> 301<https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#>
>>>>>> C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
>>>>>>
>>>>>> _http://www.bidoop.es_
>>>>>>
>>>>>>
>>>>>>
>>> --
>>> *Guillermo Ortiz*
>>> /Big Data Developer/
>>>
>>> Telf.: +34 917 680
>>> 490<https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#>
>>> Fax: +34 913 833
>>> 301<https://mail.google.com/mail/u/0/html/compose/static_files/blank_quirks.html#>
>>> C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
>>>
>>> _http://www.bidoop.es_
>>>
>>>
>
>
> --
> *Guillermo Ortiz*
> /Big Data Developer/
>
> Telf.: +34 917 680 490
> Fax: +34 913 833 301
> C/ Manuel Tovar, 49-53 - 28034 Madrid - Spain
>
> _http://www.bidoop.es_
>
