From: Jinsong Hu <jinsong...@hotmail.com>
Subject: Re: hbase doesn't delete data older than TTL in old regions
To: user@hbase.apache.org
Date: Wednesday, September 15, 2010, 11:56 AM
Hi, ryan:
I did a test with 2 key structure: 1. time:random ,
and 2. random:time.
the TTL is set to 10 minutes. the time is current system
time. the random is a random string with length 2-10
characters long.
I wrote a test program to continue to pump data into hbase
table , with the time going up with time.
for the second test case, the number of rows remains
approximately constant after it reaches
certain limit. I also checked a specific row, and wait to
20 minutes later and check it again
and found it is indeed gone.
In the first key case, the number of rows continue to grow
and the number of regions continue to grow..
to some number much higher than first case, and doesn't
stop. I checked some stores that with data
several hours old and they still remain there without
getting deleted.
Jimmy.
--------------------------------------------------
From: "Ryan Rawson" <ryano...@gmail.com>
Sent: Wednesday, September 15, 2010 11:43 AM
To: <user@hbase.apache.org>
Subject: Re: hbase doesn't delete data older than TTL in
old regions
> I feel the need to pipe in here, since people are
accusing hbase of
> having a broken feature 'TTL' when from the
description in this email
> thread, and my own knowledge doesn't really describe a
broken feature.
> Non optimal maybe, but not broken.
>
> First off, the TTL feature works on the timestamp,
thus rowkey
> structure is not related. This is because the
timestamp is stored in
> a different field. If you are also storing the
data in row key
> chronological order, then you may end up with sparse
or 'small'
> regions. But that doesn't mean the feature is
broken - ie: it does
> not remove data older than the TTL. Needs tuning
yes, but not broken.
>
> Also note that "client side deletes" work in the same
way that TTL
> does, you insert a tombstone marker, then a compaction
actually purges
> the data itself.
>
> -ryan
>
> On Wed, Sep 15, 2010 at 11:26 AM, Jinsong Hu <jinsong...@hotmail.com>
wrote:
>> I opened a ticket https://issues.apache.org/jira/browse/HBASE-2999 to
track
>> issue. dropping old store , and update the
adjacent region's key range when
>> all
>> store for a region is gone is probably the
cheapest solution, both in terms
>> of coding and in terms of resource usage in the
cluster. Do we know when
>> this can be done ?
>>
>>
>> Jimmy.
>>
>>
--------------------------------------------------
>> From: "Jonathan Gray" <jg...@facebook.com>
>> Sent: Wednesday, September 15, 2010 11:06 AM
>> To: <user@hbase.apache.org>
>> Subject: RE: hbase doesn't delete data older than
TTL in old regions
>>
>>> This sounds reasonable.
>>>
>>> We are tracking min/max timestamps in
storefiles too, so it's possible
>>> that we could expire some files of a region as
well, even if the region was
>>> not completely expired.
>>>
>>> Jinsong, mind filing a jira?
>>>
>>> JG
>>>
>>>> -----Original Message-----
>>>> From: Jinsong Hu [mailto:jinsong...@hotmail.com]
>>>> Sent: Wednesday, September 15, 2010 10:39
AM
>>>> To: user@hbase.apache.org
>>>> Subject: Re: hbase doesn't delete data
older than TTL in old regions
>>>>
>>>> Yes, Current TTL based on compaction is
working as advertised if the
>>>> key
>>>> randomly distribute the incoming data
>>>> among all regions. However, if the
key is designed in chronological
>>>> order,
>>>> the TTL doesn't really work, as no
compaction
>>>> will happen for data already written. So
we can't say that current TTL
>>>> really work as advertised, as it is key
structure dependent.
>>>>
>>>> This is a pity, because a major use case
for hbase is for people to
>>>> store
>>>> history or log data. normally people only
>>>> want to retain the data for a fixed
period. for example, US government
>>>> default data retention policy is 7 years.
Those
>>>> data are saved in chronological order.
Current TTL implementation
>>>> doesn't
>>>> work at all for those kind of use case.
>>>>
>>>> In order for that use case to really work,
hbase needs to have an
>>>> active
>>>> thread that periodically runs and check if
there
>>>> are data older than TTL, and delete the
data older than TTL is
>>>> necessary,
>>>> and compact small regions older than
certain time period
>>>> into larger ones to save system resource.
It can optimize the deletion
>>>> by
>>>> delete the whole region if it detects that
the last time
>>>> stamp for the region is older than
TTL. There should be 2 parameters
>>>> to
>>>> configure for hbase:
>>>>
>>>> 1. whether to disable/enable the TTL
thread.
>>>> 2. the interval that TTL will run. maybe
we can use a special value
>>>> like 0
>>>> to indicate that we don't run the TTL
thread, thus saving one
>>>> configuration
>>>> parameter.
>>>> for the default TTL, probably it should be
set to 1 day.
>>>> 3. How small will the region be merged. it
should be a percentage of
>>>> the
>>>> store size. for example, if 2 consecutive
region is only 10% of the
>>>> store
>>>> szie ( default is 256M), we can initiate a
region merge. We probably
>>>> need a
>>>> parameter to reduce the merge too. for
example , we only merge for
>>>> regions
>>>> who's largest timestamp
>>>> is older than half of TTL.
>>>>
>>>>
>>>> Jimmy
>>>>
>>>>
--------------------------------------------------
>>>> From: "Stack" <st...@duboce.net>
>>>> Sent: Wednesday, September 15, 2010 10:08
AM
>>>> To: <user@hbase.apache.org>
>>>> Subject: Re: hbase doesn't delete data
older than TTL in old regions
>>>>
>>>> > On Wed, Sep 15, 2010 at 9:54 AM,
Jinsong Hu <jinsong...@hotmail.com>
>>>> > wrote:
>>>> >> I have tested the TTL for hbase
and found that it relies on
>>>> compaction to
>>>> >> remove old data . However, if a
region has data that is older
>>>> >> than TTL, and there is no trigger
to compact it, then the data will
>>>> >> remain
>>>> >> there forever, wasting disk space
and memory.
>>>> >>
>>>> >
>>>> > So its working as advertised then?
>>>> >
>>>> > There's currently an issue where we
can skip major compactions if
>>>> your
>>>> > write loading has a particular
character: hbase-2990.
>>>> >
>>>> >
>>>> >> It appears at this state, to
really remove data older than TTL we
>>>> need to
>>>> >> start a client side deletion
request.
>>>> >
>>>> > Or run a manual major compaction:
>>>> >
>>>> > $ echo "major_compact TABLENAME" |
./bin/hbase shell
>>>> >
>>>> >
>>>> >
>>>> > This is really a pity because
>>>> >> it is an more expensive way to
get the job done. Another side
>>>> effect of
>>>> >> this is that as time goes on, we
will end up with some small
>>>> >> regions if the data are saved in
chronological order in regions. It
>>>> >> appears
>>>> >> that hbase doesn't have a
mechanism to merge 2 consecutive
>>>> >> small regions into a bigger one
at this time.
>>>> >
>>>> > $ ./bin/hbase
org.apache.hadoop.hbase.util.Merge
>>>> > Usage: bin/hbase merge
<table-name> <region-1> <region-2>
>>>> >
>>>> > Currently only works on offlined
table but there's a patch available
>>>> > to make it run against onlined
regions.
>>>> >
>>>> >
>>>> > So if data is saved in
>>>> >> chronological order, sooner or
later we will run out of capacity ,
>>>> even
>>>> >> if
>>>> >> the amount of data in hbase is
small, because we have lots of
>>>> regions
>>>> >> with
>>>> >> small storage space.
>>>> >>
>>>> >> A much cheaper way to remove data
older than TTL would be to
>>>> remember the
>>>> >> latest timestamp for the region
in the .META. table
>>>> >> and if the time is older than
TTL, we just adjust the row in .META.
>>>> and
>>>> >> delete the store , without doing
any compaction.
>>>> >>
>>>> >
>>>> > Say more on the above. It
sounds promising. Are you suggesting that
>>>> > in addition to compactions that we
also have a provision where we
>>>> keep
>>>> > account of a storefiles latest
timestamp (we already do this I
>>>> > believe) and that when now -
storefile-timestamp > ttl, we just
>>>> remove
>>>> > the storefile wholesale. That
sounds like it could work, if that is
>>>> > what you are suggesting. Mind
filing an issue w/ a detailed
>>>> > description?
>>>> >
>>>> > Thanks,
>>>> > St.Ack
>>>> >
>>>> >
>>>> >
>>>> >> Can this be added to the hbase
requirement for future release ?
>>>> >>
>>>> >> Jimmy
>>>> >>
>>>> >>
>>>> >>
>>>> >
>>>
>>
>
