Re: Nondeterministic outcome based on cell TTL and major compaction event order
hael_se...@hotmail.com> >>> wrote: >>> >>>> Actually I just thought of a better example… >>>> >>>> Credit Card Fraud detection. >>>> Imagine you’re being sent to work on a project out of the country. >>>> So suppose I head over across the pond and invaded Europe. ;-P >>>> >>>> I would want the credit card companies to not weigh a foreign >> transaction >>>> heavily when determining fraud, so that if they know my location is in >>>> London, then spending $$ on a dinner in London is not fraud. >>>> >>>> So I call ahead and tell my bank I’m going to be in Europe for XXX >> months.. >>>> >>>> >>>>> >>>>> As to why you would want to TTL on a column that doesn’t always use a >>>> TTL? >>>>> >>>>> I used this example in a different post… >>>>> >>>>> Imagine you have a road link which has an attribute of speed. >>>>> >>>>> You could have construction, or variable speed limits. >>>>> So you would want to change the speed limit with a TTL. >>>>> >>>>> Or you’re a retailer and you’re offering a 20% discount on a product >> for >>>> a limited time only? >>>>> >>>>> Sure, these are bad examples because in reality the database is a sync >>>> and the application would manage these type of issues. >>>>> >>>>> >>>>>> On Apr 18, 2015, at 12:23 AM, lars hofhansl wrote: >>>>>> >>>>>> The formatting did not come out right. Lemme try again... >>>>>> >>>>>> >>>>>> Just came here to say that. From our (maybe not clearly enough) >> defined >>>> semantics this how it should behave. >>>>>> >>>>>> It _is_ confusing, though, since compactions are - in a sense - just >>>> optimizations that run in the background to prevent the number of >> HFiles to >>>> be unbounded. >>>>>> In this case the schedule of the compactions influences the outcome. >>>>>> >>>>>> Note that even tombstone markers can be confusing. Here's another >>>> confusing example: >>>>>> 1. delete (r1, f1, q1, T2) >>>>>> 2. put (r1, f1, q1, v1, T1) >>>>>> >>>>>> If a compaction happens after #1 but before #2 the put will remain: >>>>>> delete >>>>>> compaction >>>>>> put (remains visible) >>>>>> >>>>>> If the compaction happens after #2 the put will be affected by the >>>> delete and hence removed: >>>>>> delete >>>>>> put >>>>>> compaction (will remove the put) >>>>>> >>>>>> Notice though that both of these examples _are_ a bit weird. >>>>>> Why would only a newer version of the cell have a TTL? >>>>>> Why would you date a delete into the future? >>>>>> >>>>>> -- Lars >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> From: lars hofhansl >>>>>> To: "dev@hbase.apache.org" >>>>>> Sent: Friday, April 17, 2015 10:18 PM >>>>>> Subject: Re: Nondeterministic outcome based on cell TTL and major >>>> compaction event order >>>>>> >>>>>> >>>>>> Just came here to say that. From our (maybe not clearly enough) >> defined >>>> semantics this how it should behave. >>>>>> >>>>>> It _is_ confusing, though, since compactions are - in a sense - just >>>> optimizations that run in the background to prevent the number of >> HFiles to >>>> be unbounded.In this case the schedule of the compactions influences the >>>> outcome. >>>>>> Note that even tombstone markers can be confusing. Here's another >>>> confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) >>>>>> If a compaction happens after #1 but before #2 the put will >>>> remain:deletecompactionput (remains visible) >>>>>> >>>>>> If the compaction happens after #2 the put will be affected by the >>&
Re: Nondeterministic outcome based on cell TTL and major compaction event order
>>>> > >>>> Note that even tombstone markers can be confusing. Here's another > >> confusing example: > >>>> 1. delete (r1, f1, q1, T2) > >>>> 2. put (r1, f1, q1, v1, T1) > >>>> > >>>> If a compaction happens after #1 but before #2 the put will remain: > >>>> delete > >>>> compaction > >>>> put (remains visible) > >>>> > >>>> If the compaction happens after #2 the put will be affected by the > >> delete and hence removed: > >>>> delete > >>>> put > >>>> compaction (will remove the put) > >>>> > >>>> Notice though that both of these examples _are_ a bit weird. > >>>> Why would only a newer version of the cell have a TTL? > >>>> Why would you date a delete into the future? > >>>> > >>>> -- Lars > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> From: lars hofhansl > >>>> To: "dev@hbase.apache.org" > >>>> Sent: Friday, April 17, 2015 10:18 PM > >>>> Subject: Re: Nondeterministic outcome based on cell TTL and major > >> compaction event order > >>>> > >>>> > >>>> Just came here to say that. From our (maybe not clearly enough) > defined > >> semantics this how it should behave. > >>>> > >>>> It _is_ confusing, though, since compactions are - in a sense - just > >> optimizations that run in the background to prevent the number of > HFiles to > >> be unbounded.In this case the schedule of the compactions influences the > >> outcome. > >>>> Note that even tombstone markers can be confusing. Here's another > >> confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) > >>>> If a compaction happens after #1 but before #2 the put will > >> remain:deletecompactionput (remains visible) > >>>> > >>>> If the compaction happens after #2 the put will be affected by the > >> delete and hence removed.deleteputcompaction (will remove the put) > >>>> > >>>> Notice though that both of these examples _are_ a bit weird.Why would > >> only a newer version of the cell have a TTL?Why would you date a delete > >> into the future? > >>>> -- Lars > >>>> > >>>>From: Sean Busbey > >>>> > >>>> > >>>> > >>>> To: dev > >>>> Sent: Friday, April 17, 2015 4:52 PM > >>>> Subject: Re: Nondeterministic outcome based on cell TTL and major > >> compaction event order > >>>> > >>>> If you have max versions set to 1 (the default), then c1 should be > >> removed > >>>> at compaction time if c2 still exists then. > >>>> > >>>> -- > >>>> Sean > >>>> > >>>> > >>>> On Apr 17, 2015 6:41 PM, "Michael Segel" > >> wrote: > >>>> > >>>>> Ok, > >>>>> So then if you have a previous cell (c1) and you insert a new cell c2 > >> that > >>>>> has a TTL of lets say 5 mins, then c1 should always exist? > >>>>> That is my understanding but from Cosmin’s post, he’s saying its > >>>>> different. And that’s why I don’t understand. You couldn’t lose the > >> cell > >>>>> c1 at all. > >>>>> Compaction or no compaction. > >>>>> > >>>>> That’s why I’m confused. Current behavior doesn’t match the expected > >>>>> contract. > >>>>> > >>>>> -Mike > >>>>> > >>>>>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell > >> wrote: > >>>>>> > >>>>>> The way TTLs work today is they define the interval of time a cell > >>>>>> exists - exactly as that. There is no tombstone laid like a normal > >>>>>> delete. Once the TTL elapses the cell just ceases to exist to normal > >>>>>> scanners. The interaction of expired cells, multiple versions, > minimum > >>>>>> versions, raw scanners, etc. can be confusing. We can absolutely > >>>>>> revisit this. > >>>>>> > >>>>>> A cell with an expired TTL could
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Yes, how you handle versioning, it will be the same regardless of the number of cell versions. (Pop from the top, drop from the bottom.) What I think would help is to define the expected outcome or contract. The code should be deterministic. That is to say, the action of writing a cell with a TTL that isn’t set to Max Long Value, should be the same each time. Then the issue of compaction should also be deterministic. There should be a small finite number of permutations based on conditional statements within the code and a single entry and exit point to the method within the class. (This gets into a coding practice that unless you’re throwing an exception, you only have one return from the code.) [Note: Its been 20 + years since I read Kernighan and Plauger … ] The idea is that we should define the outcomes and then code to it. So you can define a set of scenarios, and the defined outcome. Then take those defined scenarios / outcomes and consider compactions occuring at times during the ingestion process and then decay. Again its my understanding that the desired outcome is that if the cell decays, only that cell is affected, and will be inert (non returned ) if it exists until a major compaction removes it all together? Is this not the case? -Mike > On Apr 19, 2015, at 10:10 PM, Anoop John wrote: > > Interested example for cell level TTL Michael. > But one thing I want to say. In the above example, the versions for the > corresponding CF should have been >1.In such case there wont be issue > with major compaction right? > When versions =1 yes, it will give non deterministic results. > > -Anoop- > > > On Sun, Apr 19, 2015 at 6:59 PM, Michael Segel > wrote: > >> Actually I just thought of a better example… >> >> Credit Card Fraud detection. >> Imagine you’re being sent to work on a project out of the country. >> So suppose I head over across the pond and invaded Europe. ;-P >> >> I would want the credit card companies to not weigh a foreign transaction >> heavily when determining fraud, so that if they know my location is in >> London, then spending $$ on a dinner in London is not fraud. >> >> So I call ahead and tell my bank I’m going to be in Europe for XXX months.. >> >> >>> >>> As to why you would want to TTL on a column that doesn’t always use a >> TTL? >>> >>> I used this example in a different post… >>> >>> Imagine you have a road link which has an attribute of speed. >>> >>> You could have construction, or variable speed limits. >>> So you would want to change the speed limit with a TTL. >>> >>> Or you’re a retailer and you’re offering a 20% discount on a product for >> a limited time only? >>> >>> Sure, these are bad examples because in reality the database is a sync >> and the application would manage these type of issues. >>> >>> >>>> On Apr 18, 2015, at 12:23 AM, lars hofhansl wrote: >>>> >>>> The formatting did not come out right. Lemme try again... >>>> >>>> >>>> Just came here to say that. From our (maybe not clearly enough) defined >> semantics this how it should behave. >>>> >>>> It _is_ confusing, though, since compactions are - in a sense - just >> optimizations that run in the background to prevent the number of HFiles to >> be unbounded. >>>> In this case the schedule of the compactions influences the outcome. >>>> >>>> Note that even tombstone markers can be confusing. Here's another >> confusing example: >>>> 1. delete (r1, f1, q1, T2) >>>> 2. put (r1, f1, q1, v1, T1) >>>> >>>> If a compaction happens after #1 but before #2 the put will remain: >>>> delete >>>> compaction >>>> put (remains visible) >>>> >>>> If the compaction happens after #2 the put will be affected by the >> delete and hence removed: >>>> delete >>>> put >>>> compaction (will remove the put) >>>> >>>> Notice though that both of these examples _are_ a bit weird. >>>> Why would only a newer version of the cell have a TTL? >>>> Why would you date a delete into the future? >>>> >>>> -- Lars >>>> >>>> >>>> >>>> >>>> >>>> From: lars hofhansl >>>> To: "dev@hbase.apache.org" >>>> Sent: Friday, April 17, 2015 10:18 PM >>>> Subject: Re: Nondeterminist
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Interested example for cell level TTL Michael. But one thing I want to say. In the above example, the versions for the corresponding CF should have been >1.In such case there wont be issue with major compaction right? When versions =1 yes, it will give non deterministic results. -Anoop- On Sun, Apr 19, 2015 at 6:59 PM, Michael Segel wrote: > Actually I just thought of a better example… > > Credit Card Fraud detection. > Imagine you’re being sent to work on a project out of the country. > So suppose I head over across the pond and invaded Europe. ;-P > > I would want the credit card companies to not weigh a foreign transaction > heavily when determining fraud, so that if they know my location is in > London, then spending $$ on a dinner in London is not fraud. > > So I call ahead and tell my bank I’m going to be in Europe for XXX months.. > > > > > > As to why you would want to TTL on a column that doesn’t always use a > TTL? > > > > I used this example in a different post… > > > > Imagine you have a road link which has an attribute of speed. > > > > You could have construction, or variable speed limits. > > So you would want to change the speed limit with a TTL. > > > > Or you’re a retailer and you’re offering a 20% discount on a product for > a limited time only? > > > > Sure, these are bad examples because in reality the database is a sync > and the application would manage these type of issues. > > > > > >> On Apr 18, 2015, at 12:23 AM, lars hofhansl wrote: > >> > >> The formatting did not come out right. Lemme try again... > >> > >> > >> Just came here to say that. From our (maybe not clearly enough) defined > semantics this how it should behave. > >> > >> It _is_ confusing, though, since compactions are - in a sense - just > optimizations that run in the background to prevent the number of HFiles to > be unbounded. > >> In this case the schedule of the compactions influences the outcome. > >> > >> Note that even tombstone markers can be confusing. Here's another > confusing example: > >> 1. delete (r1, f1, q1, T2) > >> 2. put (r1, f1, q1, v1, T1) > >> > >> If a compaction happens after #1 but before #2 the put will remain: > >> delete > >> compaction > >> put (remains visible) > >> > >> If the compaction happens after #2 the put will be affected by the > delete and hence removed: > >> delete > >> put > >> compaction (will remove the put) > >> > >> Notice though that both of these examples _are_ a bit weird. > >> Why would only a newer version of the cell have a TTL? > >> Why would you date a delete into the future? > >> > >> -- Lars > >> > >> > >> > >> > >> > >> From: lars hofhansl > >> To: "dev@hbase.apache.org" > >> Sent: Friday, April 17, 2015 10:18 PM > >> Subject: Re: Nondeterministic outcome based on cell TTL and major > compaction event order > >> > >> > >> Just came here to say that. From our (maybe not clearly enough) defined > semantics this how it should behave. > >> > >> It _is_ confusing, though, since compactions are - in a sense - just > optimizations that run in the background to prevent the number of HFiles to > be unbounded.In this case the schedule of the compactions influences the > outcome. > >> Note that even tombstone markers can be confusing. Here's another > confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) > >> If a compaction happens after #1 but before #2 the put will > remain:deletecompactionput (remains visible) > >> > >> If the compaction happens after #2 the put will be affected by the > delete and hence removed.deleteputcompaction (will remove the put) > >> > >> Notice though that both of these examples _are_ a bit weird.Why would > only a newer version of the cell have a TTL?Why would you date a delete > into the future? > >> -- Lars > >> > >> From: Sean Busbey > >> > >> > >> > >> To: dev > >> Sent: Friday, April 17, 2015 4:52 PM > >> Subject: Re: Nondeterministic outcome based on cell TTL and major > compaction event order > >> > >> If you have max versions set to 1 (the default), then c1 should be > removed > >> at compaction time if c2 still exists then. > >> > >> -- > >> Sean > >> > >> > >>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Actually I just thought of a better example… Credit Card Fraud detection. Imagine you’re being sent to work on a project out of the country. So suppose I head over across the pond and invaded Europe. ;-P I would want the credit card companies to not weigh a foreign transaction heavily when determining fraud, so that if they know my location is in London, then spending $$ on a dinner in London is not fraud. So I call ahead and tell my bank I’m going to be in Europe for XXX months.. > > As to why you would want to TTL on a column that doesn’t always use a TTL? > > I used this example in a different post… > > Imagine you have a road link which has an attribute of speed. > > You could have construction, or variable speed limits. > So you would want to change the speed limit with a TTL. > > Or you’re a retailer and you’re offering a 20% discount on a product for a > limited time only? > > Sure, these are bad examples because in reality the database is a sync and > the application would manage these type of issues. > > >> On Apr 18, 2015, at 12:23 AM, lars hofhansl wrote: >> >> The formatting did not come out right. Lemme try again... >> >> >> Just came here to say that. From our (maybe not clearly enough) defined >> semantics this how it should behave. >> >> It _is_ confusing, though, since compactions are - in a sense - just >> optimizations that run in the background to prevent the number of HFiles to >> be unbounded. >> In this case the schedule of the compactions influences the outcome. >> >> Note that even tombstone markers can be confusing. Here's another confusing >> example: >> 1. delete (r1, f1, q1, T2) >> 2. put (r1, f1, q1, v1, T1) >> >> If a compaction happens after #1 but before #2 the put will remain: >> delete >> compaction >> put (remains visible) >> >> If the compaction happens after #2 the put will be affected by the delete >> and hence removed: >> delete >> put >> compaction (will remove the put) >> >> Notice though that both of these examples _are_ a bit weird. >> Why would only a newer version of the cell have a TTL? >> Why would you date a delete into the future? >> >> -- Lars >> >> >> >> >> >> From: lars hofhansl >> To: "dev@hbase.apache.org" >> Sent: Friday, April 17, 2015 10:18 PM >> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction >> event order >> >> >> Just came here to say that. From our (maybe not clearly enough) defined >> semantics this how it should behave. >> >> It _is_ confusing, though, since compactions are - in a sense - just >> optimizations that run in the background to prevent the number of HFiles to >> be unbounded.In this case the schedule of the compactions influences the >> outcome. >> Note that even tombstone markers can be confusing. Here's another confusing >> example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) >> If a compaction happens after #1 but before #2 the put will >> remain:deletecompactionput (remains visible) >> >> If the compaction happens after #2 the put will be affected by the delete >> and hence removed.deleteputcompaction (will remove the put) >> >> Notice though that both of these examples _are_ a bit weird.Why would only a >> newer version of the cell have a TTL?Why would you date a delete into the >> future? >> -- Lars >> >> From: Sean Busbey >> >> >> >> To: dev >> Sent: Friday, April 17, 2015 4:52 PM >> Subject: Re: Nondeterministic outcome based on cell TTL and major compaction >> event order >> >> If you have max versions set to 1 (the default), then c1 should be removed >> at compaction time if c2 still exists then. >> >> -- >> Sean >> >> >> On Apr 17, 2015 6:41 PM, "Michael Segel" wrote: >> >>> Ok, >>> So then if you have a previous cell (c1) and you insert a new cell c2 that >>> has a TTL of lets say 5 mins, then c1 should always exist? >>> That is my understanding but from Cosmin’s post, he’s saying its >>> different. And that’s why I don’t understand. You couldn’t lose the cell >>> c1 at all. >>> Compaction or no compaction. >>> >>> That’s why I’m confused. Current behavior doesn’t match the expected >>> contract. >>> >>> -Mike >>> >>>> On Apr 17, 2015, at 4:37 PM, Andrew
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Ok… In your example the put has an earlier time T1 than T2? I’m assuming that the orders are reversed due to something in the memcache? Also what happens to the data within the memcache that hasn’t been written to file and you insert a delete after a put() ? As to why you would want to TTL on a column that doesn’t always use a TTL? I used this example in a different post… Imagine you have a road link which has an attribute of speed. You could have construction, or variable speed limits. So you would want to change the speed limit with a TTL. Or you’re a retailer and you’re offering a 20% discount on a product for a limited time only? Sure, these are bad examples because in reality the database is a sync and the application would manage these type of issues. > On Apr 18, 2015, at 12:23 AM, lars hofhansl wrote: > > The formatting did not come out right. Lemme try again... > > > Just came here to say that. From our (maybe not clearly enough) defined > semantics this how it should behave. > > It _is_ confusing, though, since compactions are - in a sense - just > optimizations that run in the background to prevent the number of HFiles to > be unbounded. > In this case the schedule of the compactions influences the outcome. > > Note that even tombstone markers can be confusing. Here's another confusing > example: > 1. delete (r1, f1, q1, T2) > 2. put (r1, f1, q1, v1, T1) > > If a compaction happens after #1 but before #2 the put will remain: > delete > compaction > put (remains visible) > > If the compaction happens after #2 the put will be affected by the delete and > hence removed: > delete > put > compaction (will remove the put) > > Notice though that both of these examples _are_ a bit weird. > Why would only a newer version of the cell have a TTL? > Why would you date a delete into the future? > > -- Lars > > > > > ____ > From: lars hofhansl > To: "dev@hbase.apache.org" > Sent: Friday, April 17, 2015 10:18 PM > Subject: Re: Nondeterministic outcome based on cell TTL and major compaction > event order > > > Just came here to say that. From our (maybe not clearly enough) defined > semantics this how it should behave. > > It _is_ confusing, though, since compactions are - in a sense - just > optimizations that run in the background to prevent the number of HFiles to > be unbounded.In this case the schedule of the compactions influences the > outcome. > Note that even tombstone markers can be confusing. Here's another confusing > example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) > If a compaction happens after #1 but before #2 the put will > remain:deletecompactionput (remains visible) > > If the compaction happens after #2 the put will be affected by the delete and > hence removed.deleteputcompaction (will remove the put) > > Notice though that both of these examples _are_ a bit weird.Why would only a > newer version of the cell have a TTL?Why would you date a delete into the > future? > -- Lars > > From: Sean Busbey > > > > To: dev > Sent: Friday, April 17, 2015 4:52 PM > Subject: Re: Nondeterministic outcome based on cell TTL and major compaction > event order > > If you have max versions set to 1 (the default), then c1 should be removed > at compaction time if c2 still exists then. > > -- > Sean > > > On Apr 17, 2015 6:41 PM, "Michael Segel" wrote: > >> Ok, >> So then if you have a previous cell (c1) and you insert a new cell c2 that >> has a TTL of lets say 5 mins, then c1 should always exist? >> That is my understanding but from Cosmin’s post, he’s saying its >> different. And that’s why I don’t understand. You couldn’t lose the cell >> c1 at all. >> Compaction or no compaction. >> >> That’s why I’m confused. Current behavior doesn’t match the expected >> contract. >> >> -Mike >> >>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell wrote: >>> >>> The way TTLs work today is they define the interval of time a cell >>> exists - exactly as that. There is no tombstone laid like a normal >>> delete. Once the TTL elapses the cell just ceases to exist to normal >>> scanners. The interaction of expired cells, multiple versions, minimum >>> versions, raw scanners, etc. can be confusing. We can absolutely >>> revisit this. >>> >>> A cell with an expired TTL could be treated as the combination of >>> tombstone and the most recent value it lays over. This is not how the >>> implementation works today, but could be changed for an upco
Re: Nondeterministic outcome based on cell TTL and major compaction event order
I said barring max versions… (in an earlier post on the thread.) > On Apr 17, 2015, at 6:52 PM, Sean Busbey wrote: > > If you have max versions set to 1 (the default), then c1 should be removed > at compaction time if c2 still exists then. > > -- > Sean > On Apr 17, 2015 6:41 PM, "Michael Segel" wrote: > >> Ok, >> So then if you have a previous cell (c1) and you insert a new cell c2 that >> has a TTL of lets say 5 mins, then c1 should always exist? >> That is my understanding but from Cosmin’s post, he’s saying its >> different. And that’s why I don’t understand. You couldn’t lose the cell >> c1 at all. >> Compaction or no compaction. >> >> That’s why I’m confused. Current behavior doesn’t match the expected >> contract. >> >> -Mike >> >>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell wrote: >>> >>> The way TTLs work today is they define the interval of time a cell >>> exists - exactly as that. There is no tombstone laid like a normal >>> delete. Once the TTL elapses the cell just ceases to exist to normal >>> scanners. The interaction of expired cells, multiple versions, minimum >>> versions, raw scanners, etc. can be confusing. We can absolutely >>> revisit this. >>> >>> A cell with an expired TTL could be treated as the combination of >>> tombstone and the most recent value it lays over. This is not how the >>> implementation works today, but could be changed for an upcoming major >>> version like 2.0 if there's consensus to do it. >>> >>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene wrote: I've been initially puzzled by this, although I realize how it's likely >> as designed. The cell TTL expiration and compactions events can lead to either some >> (the older) data left or no data at all for a particular (row, family, >> qualifier, ts) coordinate. Write (r1, f1, q1, v1, 1) Write (r1, f1, q1, v1, 2) - TTL=1 minute Scenario 1: If a major compaction happens within a minute it will remove (r1, f1, q1, v1, 1) then after a minute (r1, f1, q1, v1, 2) will expire no data left Scenario 2: A minute passes (r1, f1, q1, v1, 2) expires Compaction runs.. (r1, f1, q1, v1, 1) remains This seems, by and large expected behavior, but it still seems >> "uncomfortable" that the (overall) outcome is not decided by me, but by a >> chance of event ordering. I wonder we'd want this to behave differently (perhaps it has been >> discussed already), but if not, it's worth a more detailed documentation in >> the book. What do you think? Cosmin >>> >>> -- >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet >>> Hein (via Tom White) >>> >> >> The opinions expressed here are mine, while they may reflect a cognitive >> thought, that is purely accidental. >> Use at your own risk. >> Michael Segel >> michael_segel (AT) hotmail.com >> >> >> >> >> >> The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction event order
The formatting did not come out right. Lemme try again... Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave. It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded. In this case the schedule of the compactions influences the outcome. Note that even tombstone markers can be confusing. Here's another confusing example: 1. delete (r1, f1, q1, T2) 2. put (r1, f1, q1, v1, T1) If a compaction happens after #1 but before #2 the put will remain: delete compaction put (remains visible) If the compaction happens after #2 the put will be affected by the delete and hence removed: delete put compaction (will remove the put) Notice though that both of these examples _are_ a bit weird. Why would only a newer version of the cell have a TTL? Why would you date a delete into the future? -- Lars From: lars hofhansl To: "dev@hbase.apache.org" Sent: Friday, April 17, 2015 10:18 PM Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave. It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.In this case the schedule of the compactions influences the outcome. Note that even tombstone markers can be confusing. Here's another confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) If a compaction happens after #1 but before #2 the put will remain:deletecompactionput (remains visible) If the compaction happens after #2 the put will be affected by the delete and hence removed.deleteputcompaction (will remove the put) Notice though that both of these examples _are_ a bit weird.Why would only a newer version of the cell have a TTL?Why would you date a delete into the future? -- Lars From: Sean Busbey To: dev Sent: Friday, April 17, 2015 4:52 PM Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order If you have max versions set to 1 (the default), then c1 should be removed at compaction time if c2 still exists then. -- Sean On Apr 17, 2015 6:41 PM, "Michael Segel" wrote: > Ok, > So then if you have a previous cell (c1) and you insert a new cell c2 that > has a TTL of lets say 5 mins, then c1 should always exist? > That is my understanding but from Cosmin’s post, he’s saying its > different. And that’s why I don’t understand. You couldn’t lose the cell > c1 at all. > Compaction or no compaction. > > That’s why I’m confused. Current behavior doesn’t match the expected > contract. > > -Mike > > > On Apr 17, 2015, at 4:37 PM, Andrew Purtell wrote: > > > > The way TTLs work today is they define the interval of time a cell > > exists - exactly as that. There is no tombstone laid like a normal > > delete. Once the TTL elapses the cell just ceases to exist to normal > > scanners. The interaction of expired cells, multiple versions, minimum > > versions, raw scanners, etc. can be confusing. We can absolutely > > revisit this. > > > > A cell with an expired TTL could be treated as the combination of > > tombstone and the most recent value it lays over. This is not how the > > implementation works today, but could be changed for an upcoming major > > version like 2.0 if there's consensus to do it. > > > > > >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene wrote: > >> > >> I've been initially puzzled by this, although I realize how it's likely > as designed. > >> > >> > >> The cell TTL expiration and compactions events can lead to either some > (the older) data left or no data at all for a particular (row, family, > qualifier, ts) coordinate. > >> > >> > >> > >> Write (r1, f1, q1, v1, 1) > >> > >> Write (r1, f1, q1, v1, 2) - TTL=1 minute > >> > >> > >> Scenario 1: > >> > >> > >> If a major compaction happens within a minute > >> > >> > >> it will remove (r1, f1, q1, v1, 1) > >> > >> then after a minute (r1, f1, q1, v1, 2) will expire > >> > >> no data left > >> > >> > >> Scenario 2: > >> > >> > >> A minute passes > >> > >> (r1, f1, q1, v1, 2) expires > >> > >> Compaction runs.. > >> > >> (r1, f1, q1, v1, 1) remains > >> > >> > >> > >> This seems, by and large expe
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Just came here to say that. From our (maybe not clearly enough) defined semantics this how it should behave. It _is_ confusing, though, since compactions are - in a sense - just optimizations that run in the background to prevent the number of HFiles to be unbounded.In this case the schedule of the compactions influences the outcome. Note that even tombstone markers can be confusing. Here's another confusing example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) If a compaction happens after #1 but before #2 the put will remain:deletecompactionput (remains visible) If the compaction happens after #2 the put will be affected by the delete and hence removed.deleteputcompaction (will remove the put) Notice though that both of these examples _are_ a bit weird.Why would only a newer version of the cell have a TTL?Why would you date a delete into the future? -- Lars From: Sean Busbey To: dev Sent: Friday, April 17, 2015 4:52 PM Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order If you have max versions set to 1 (the default), then c1 should be removed at compaction time if c2 still exists then. -- Sean On Apr 17, 2015 6:41 PM, "Michael Segel" wrote: > Ok, > So then if you have a previous cell (c1) and you insert a new cell c2 that > has a TTL of lets say 5 mins, then c1 should always exist? > That is my understanding but from Cosmin’s post, he’s saying its > different. And that’s why I don’t understand. You couldn’t lose the cell > c1 at all. > Compaction or no compaction. > > That’s why I’m confused. Current behavior doesn’t match the expected > contract. > > -Mike > > > On Apr 17, 2015, at 4:37 PM, Andrew Purtell wrote: > > > > The way TTLs work today is they define the interval of time a cell > > exists - exactly as that. There is no tombstone laid like a normal > > delete. Once the TTL elapses the cell just ceases to exist to normal > > scanners. The interaction of expired cells, multiple versions, minimum > > versions, raw scanners, etc. can be confusing. We can absolutely > > revisit this. > > > > A cell with an expired TTL could be treated as the combination of > > tombstone and the most recent value it lays over. This is not how the > > implementation works today, but could be changed for an upcoming major > > version like 2.0 if there's consensus to do it. > > > > > >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene wrote: > >> > >> I've been initially puzzled by this, although I realize how it's likely > as designed. > >> > >> > >> The cell TTL expiration and compactions events can lead to either some > (the older) data left or no data at all for a particular (row, family, > qualifier, ts) coordinate. > >> > >> > >> > >> Write (r1, f1, q1, v1, 1) > >> > >> Write (r1, f1, q1, v1, 2) - TTL=1 minute > >> > >> > >> Scenario 1: > >> > >> > >> If a major compaction happens within a minute > >> > >> > >> it will remove (r1, f1, q1, v1, 1) > >> > >> then after a minute (r1, f1, q1, v1, 2) will expire > >> > >> no data left > >> > >> > >> Scenario 2: > >> > >> > >> A minute passes > >> > >> (r1, f1, q1, v1, 2) expires > >> > >> Compaction runs.. > >> > >> (r1, f1, q1, v1, 1) remains > >> > >> > >> > >> This seems, by and large expected behavior, but it still seems > "uncomfortable" that the (overall) outcome is not decided by me, but by a > chance of event ordering. > >> > >> > >> I wonder we'd want this to behave differently (perhaps it has been > discussed already), but if not, it's worth a more detailed documentation in > the book. > >> > >> > >> What do you think? > >> > >> > >> Cosmin > >> > >> > >> > >> > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein (via Tom White) > > > > The opinions expressed here are mine, while they may reflect a cognitive > thought, that is purely accidental. > Use at your own risk. > Michael Segel > michael_segel (AT) hotmail.com > > > > > >
Re: Nondeterministic outcome based on cell TTL and major compaction event order
If you have max versions set to 1 (the default), then c1 should be removed at compaction time if c2 still exists then. -- Sean On Apr 17, 2015 6:41 PM, "Michael Segel" wrote: > Ok, > So then if you have a previous cell (c1) and you insert a new cell c2 that > has a TTL of lets say 5 mins, then c1 should always exist? > That is my understanding but from Cosmin’s post, he’s saying its > different. And that’s why I don’t understand. You couldn’t lose the cell > c1 at all. > Compaction or no compaction. > > That’s why I’m confused. Current behavior doesn’t match the expected > contract. > > -Mike > > > On Apr 17, 2015, at 4:37 PM, Andrew Purtell wrote: > > > > The way TTLs work today is they define the interval of time a cell > > exists - exactly as that. There is no tombstone laid like a normal > > delete. Once the TTL elapses the cell just ceases to exist to normal > > scanners. The interaction of expired cells, multiple versions, minimum > > versions, raw scanners, etc. can be confusing. We can absolutely > > revisit this. > > > > A cell with an expired TTL could be treated as the combination of > > tombstone and the most recent value it lays over. This is not how the > > implementation works today, but could be changed for an upcoming major > > version like 2.0 if there's consensus to do it. > > > > > >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene wrote: > >> > >> I've been initially puzzled by this, although I realize how it's likely > as designed. > >> > >> > >> The cell TTL expiration and compactions events can lead to either some > (the older) data left or no data at all for a particular (row, family, > qualifier, ts) coordinate. > >> > >> > >> > >> Write (r1, f1, q1, v1, 1) > >> > >> Write (r1, f1, q1, v1, 2) - TTL=1 minute > >> > >> > >> Scenario 1: > >> > >> > >> If a major compaction happens within a minute > >> > >> > >> it will remove (r1, f1, q1, v1, 1) > >> > >> then after a minute (r1, f1, q1, v1, 2) will expire > >> > >> no data left > >> > >> > >> Scenario 2: > >> > >> > >> A minute passes > >> > >> (r1, f1, q1, v1, 2) expires > >> > >> Compaction runs.. > >> > >> (r1, f1, q1, v1, 1) remains > >> > >> > >> > >> This seems, by and large expected behavior, but it still seems > "uncomfortable" that the (overall) outcome is not decided by me, but by a > chance of event ordering. > >> > >> > >> I wonder we'd want this to behave differently (perhaps it has been > discussed already), but if not, it's worth a more detailed documentation in > the book. > >> > >> > >> What do you think? > >> > >> > >> Cosmin > >> > >> > >> > >> > > > > -- > > Best regards, > > > > - Andy > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > Hein (via Tom White) > > > > The opinions expressed here are mine, while they may reflect a cognitive > thought, that is purely accidental. > Use at your own risk. > Michael Segel > michael_segel (AT) hotmail.com > > > > > >
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Ok, So then if you have a previous cell (c1) and you insert a new cell c2 that has a TTL of lets say 5 mins, then c1 should always exist? That is my understanding but from Cosmin’s post, he’s saying its different. And that’s why I don’t understand. You couldn’t lose the cell c1 at all. Compaction or no compaction. That’s why I’m confused. Current behavior doesn’t match the expected contract. -Mike > On Apr 17, 2015, at 4:37 PM, Andrew Purtell wrote: > > The way TTLs work today is they define the interval of time a cell > exists - exactly as that. There is no tombstone laid like a normal > delete. Once the TTL elapses the cell just ceases to exist to normal > scanners. The interaction of expired cells, multiple versions, minimum > versions, raw scanners, etc. can be confusing. We can absolutely > revisit this. > > A cell with an expired TTL could be treated as the combination of > tombstone and the most recent value it lays over. This is not how the > implementation works today, but could be changed for an upcoming major > version like 2.0 if there's consensus to do it. > > >> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene wrote: >> >> I've been initially puzzled by this, although I realize how it's likely as >> designed. >> >> >> The cell TTL expiration and compactions events can lead to either some (the >> older) data left or no data at all for a particular (row, family, >> qualifier, ts) coordinate. >> >> >> >> Write (r1, f1, q1, v1, 1) >> >> Write (r1, f1, q1, v1, 2) - TTL=1 minute >> >> >> Scenario 1: >> >> >> If a major compaction happens within a minute >> >> >> it will remove (r1, f1, q1, v1, 1) >> >> then after a minute (r1, f1, q1, v1, 2) will expire >> >> no data left >> >> >> Scenario 2: >> >> >> A minute passes >> >> (r1, f1, q1, v1, 2) expires >> >> Compaction runs.. >> >> (r1, f1, q1, v1, 1) remains >> >> >> >> This seems, by and large expected behavior, but it still seems >> "uncomfortable" that the (overall) outcome is not decided by me, but by a >> chance of event ordering. >> >> >> I wonder we'd want this to behave differently (perhaps it has been discussed >> already), but if not, it's worth a more detailed documentation in the book. >> >> >> What do you think? >> >> >> Cosmin >> >> >> >> > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet > Hein (via Tom White) > The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction event order
The way TTLs work today is they define the interval of time a cell exists - exactly as that. There is no tombstone laid like a normal delete. Once the TTL elapses the cell just ceases to exist to normal scanners. The interaction of expired cells, multiple versions, minimum versions, raw scanners, etc. can be confusing. We can absolutely revisit this. A cell with an expired TTL could be treated as the combination of tombstone and the most recent value it lays over. This is not how the implementation works today, but could be changed for an upcoming major version like 2.0 if there's consensus to do it. > On Apr 10, 2015, at 7:26 AM, Cosmin Lehene wrote: > > I've been initially puzzled by this, although I realize how it's likely as > designed. > > > The cell TTL expiration and compactions events can lead to either some (the > older) data left or no data at all for a particular (row, family, qualifier, > ts) coordinate. > > > > Write (r1, f1, q1, v1, 1) > > Write (r1, f1, q1, v1, 2) - TTL=1 minute > > > Scenario 1: > > > If a major compaction happens within a minute > > > it will remove (r1, f1, q1, v1, 1) > > then after a minute (r1, f1, q1, v1, 2) will expire > > no data left > > > Scenario 2: > > > A minute passes > > (r1, f1, q1, v1, 2) expires > > Compaction runs.. > > (r1, f1, q1, v1, 1) remains > > > > This seems, by and large expected behavior, but it still seems > "uncomfortable" that the (overall) outcome is not decided by me, but by a > chance of event ordering. > > > I wonder we'd want this to behave differently (perhaps it has been discussed > already), but if not, it's worth a more detailed documentation in the book. > > > What do you think? > > > Cosmin > > > > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
Re: Nondeterministic outcome based on cell TTL and major compaction event order
No, you need to count them. The exist therefore they count. ;-) But that doesn’t answer the question about the expected behavior of TTL. > On Apr 17, 2015, at 3:09 PM, Jean-Marc Spaggiari > wrote: > > Maybe we should just not counts the cells with TTL when counting for the > versions? That way, what ever we do we always keep the initial value? > > Better to have an extra version and have a consistent behaviour I think... > > JMS > Le 2015-04-17 15:28, "Michael Segel" a écrit : > >> I think you misunderstood the question(s). >> >> How do you define the meaning of the TTL attribute? >> >> Is the TTL a special form of the delete tombstone? >> >> The reason I ask this is that if we look at your example you’ve stated >> that if a compaction occurred prior to the end of life of cell 2, cell 1 >> would be destroyed. >> That doesn’t make sense as an expected result. >> >> This is why I am asking what is the defined contract of the TTL feature. >> >> Cell 1 should never be deleted. (unless you explicitly state the number of >> versions to be 1 then it would be marked for deletion when you insert cell >> 2.) >> >> Thx >> >> -Mike >> >> >>> On Apr 13, 2015, at 6:12 PM, Cosmin Lehene wrote: >>> >>> The ambiguity seems to lie at the intersection of TTL and version >> "garbage collection" during compactions. >>> >>> Major compactions can lead to nondeterministic results when multiple >> versions are involved (slightly captured in the book >> http://hbase.apache.org/book.html#versions and >> http://www.ngdata.com/bending-time-in-hbase/ ) >>> TTL expirations don't result in deletes (at least not in the classical >> sense with a tombstone). >>> >>> Cosmin >>> >>> ___ >>> From: Michael Segel >>> Sent: Friday, April 10, 2015 8:35 AM >>> To: dev@hbase.apache.org >>> Subject: Re: Nondeterministic outcome based on cell TTL and major >> compaction event order >>> >>> Interesting. >>> There seems to be some ambiguity in what happens between a TTL and a >> deletion. >>> >>> Is the TTL a delete or is it a separate type of function? >>> >>> That is to say when you inserted version 2 of the cell, did you intend >> to just have version 2 exist for a little while and then default to version >> 1 or did you mean that when you inserted version 2, you wanted to delete >> everything prior to version 2 and then when version 2 expires, it then goes >> away? >>> >>> The documentation isn’t clear on this point. >>> >>> To give you an example where you wouldn’t want to have the TTL on a cell >> also delete prior versions… >>> >>> Suppose you’re storing map data in HBase. You have an attribute (speed) >> associated to a road link. >>> >>> If the road is a 65 MPH highway, then the base speed (default speed) is >> 65MPH. However if there’s construction planned for the road then you need >> to reset the speed to 45 mph while there is construction. You know that >> the construction is supposed to last X months, so you reset the speed limit >> to 45 with a TTL on that cell version only. >>> >>> Another example is if you’re storing price for a given sku in a given >> region of your retail chain. So you want to reduce the price by 20% for a >> 2 week period. >>> Again, you set that discount to live for 2 weeks with a TTL, then revert >> back to original price. >>> >>> So I guess there should be a clarification as to what is intended for >> the TTL to do? >>> >>> Does that make sense? >>> >>> >>> >>> >>> >>>> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene wrote: >>>> >>>> I've been initially puzzled by this, although I realize how it's likely >> as designed. >>>> >>>> >>>> The cell TTL expiration and compactions events can lead to either some >> (the older) data left or no data at all for a particular (row, family, >> qualifier, ts) coordinate. >>>> >>>> >>>> >>>> Write (r1, f1, q1, v1, 1) >>>> >>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute >>>> >>>> >>>> Scenario 1: >>>> >>>> >>>> If a major compaction happens within a minute >>>>
Re: Nondeterministic outcome based on cell TTL and major compaction event order
Maybe we should just not counts the cells with TTL when counting for the versions? That way, what ever we do we always keep the initial value? Better to have an extra version and have a consistent behaviour I think... JMS Le 2015-04-17 15:28, "Michael Segel" a écrit : > I think you misunderstood the question(s). > > How do you define the meaning of the TTL attribute? > > Is the TTL a special form of the delete tombstone? > > The reason I ask this is that if we look at your example you’ve stated > that if a compaction occurred prior to the end of life of cell 2, cell 1 > would be destroyed. > That doesn’t make sense as an expected result. > > This is why I am asking what is the defined contract of the TTL feature. > > Cell 1 should never be deleted. (unless you explicitly state the number of > versions to be 1 then it would be marked for deletion when you insert cell > 2.) > > Thx > > -Mike > > > > On Apr 13, 2015, at 6:12 PM, Cosmin Lehene wrote: > > > > The ambiguity seems to lie at the intersection of TTL and version > "garbage collection" during compactions. > > > > Major compactions can lead to nondeterministic results when multiple > versions are involved (slightly captured in the book > http://hbase.apache.org/book.html#versions and > http://www.ngdata.com/bending-time-in-hbase/ ) > > TTL expirations don't result in deletes (at least not in the classical > sense with a tombstone). > > > > Cosmin > > > > _______________________ > > From: Michael Segel > > Sent: Friday, April 10, 2015 8:35 AM > > To: dev@hbase.apache.org > > Subject: Re: Nondeterministic outcome based on cell TTL and major > compaction event order > > > > Interesting. > > There seems to be some ambiguity in what happens between a TTL and a > deletion. > > > > Is the TTL a delete or is it a separate type of function? > > > > That is to say when you inserted version 2 of the cell, did you intend > to just have version 2 exist for a little while and then default to version > 1 or did you mean that when you inserted version 2, you wanted to delete > everything prior to version 2 and then when version 2 expires, it then goes > away? > > > > The documentation isn’t clear on this point. > > > > To give you an example where you wouldn’t want to have the TTL on a cell > also delete prior versions… > > > > Suppose you’re storing map data in HBase. You have an attribute (speed) > associated to a road link. > > > > If the road is a 65 MPH highway, then the base speed (default speed) is > 65MPH. However if there’s construction planned for the road then you need > to reset the speed to 45 mph while there is construction. You know that > the construction is supposed to last X months, so you reset the speed limit > to 45 with a TTL on that cell version only. > > > > Another example is if you’re storing price for a given sku in a given > region of your retail chain. So you want to reduce the price by 20% for a > 2 week period. > > Again, you set that discount to live for 2 weeks with a TTL, then revert > back to original price. > > > > So I guess there should be a clarification as to what is intended for > the TTL to do? > > > > Does that make sense? > > > > > > > > > > > >> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene wrote: > >> > >> I've been initially puzzled by this, although I realize how it's likely > as designed. > >> > >> > >> The cell TTL expiration and compactions events can lead to either some > (the older) data left or no data at all for a particular (row, family, > qualifier, ts) coordinate. > >> > >> > >> > >> Write (r1, f1, q1, v1, 1) > >> > >> Write (r1, f1, q1, v1, 2) - TTL=1 minute > >> > >> > >> Scenario 1: > >> > >> > >> If a major compaction happens within a minute > >> > >> > >> it will remove (r1, f1, q1, v1, 1) > >> > >> then after a minute (r1, f1, q1, v1, 2) will expire > >> > >> no data left > >> > >> > >> Scenario 2: > >> > >> > >> A minute passes > >> > >> (r1, f1, q1, v1, 2) expires > >> > >> Compaction runs.. > >> > >> (r1, f1, q1, v1, 1) remains > >> > >> > >> > >> This seems, by and large expected behavior, but it still seems > "uncomfortable" that the (overall) outcome is not decided by me, but by a > chance of event ordering. > >> > >> > >> I wonder we'd want this to behave differently (perhaps it has been > discussed already), but if not, it's worth a more detailed documentation in > the book. > >> > >> > >> What do you think? > >> > >> > >> Cosmin > >> > >> > >> > >> > > > > The opinions expressed here are mine, while they may reflect a cognitive > thought, that is purely accidental. > > Use at your own risk. > > Michael Segel > > michael_segel (AT) hotmail.com > > > > > > > > > > > > > > The opinions expressed here are mine, while they may reflect a cognitive > thought, that is purely accidental. > Use at your own risk. > Michael Segel > michael_segel (AT) hotmail.com > > > > > >
Re: Nondeterministic outcome based on cell TTL and major compaction event order
I think you misunderstood the question(s). How do you define the meaning of the TTL attribute? Is the TTL a special form of the delete tombstone? The reason I ask this is that if we look at your example you’ve stated that if a compaction occurred prior to the end of life of cell 2, cell 1 would be destroyed. That doesn’t make sense as an expected result. This is why I am asking what is the defined contract of the TTL feature. Cell 1 should never be deleted. (unless you explicitly state the number of versions to be 1 then it would be marked for deletion when you insert cell 2.) Thx -Mike > On Apr 13, 2015, at 6:12 PM, Cosmin Lehene wrote: > > The ambiguity seems to lie at the intersection of TTL and version "garbage > collection" during compactions. > > Major compactions can lead to nondeterministic results when multiple versions > are involved (slightly captured in the book > http://hbase.apache.org/book.html#versions and > http://www.ngdata.com/bending-time-in-hbase/ ) > TTL expirations don't result in deletes (at least not in the classical sense > with a tombstone). > > Cosmin > > ___ > From: Michael Segel > Sent: Friday, April 10, 2015 8:35 AM > To: dev@hbase.apache.org > Subject: Re: Nondeterministic outcome based on cell TTL and major compaction > event order > > Interesting. > There seems to be some ambiguity in what happens between a TTL and a deletion. > > Is the TTL a delete or is it a separate type of function? > > That is to say when you inserted version 2 of the cell, did you intend to > just have version 2 exist for a little while and then default to version 1 or > did you mean that when you inserted version 2, you wanted to delete > everything prior to version 2 and then when version 2 expires, it then goes > away? > > The documentation isn’t clear on this point. > > To give you an example where you wouldn’t want to have the TTL on a cell also > delete prior versions… > > Suppose you’re storing map data in HBase. You have an attribute (speed) > associated to a road link. > > If the road is a 65 MPH highway, then the base speed (default speed) is > 65MPH. However if there’s construction planned for the road then you need to > reset the speed to 45 mph while there is construction. You know that the > construction is supposed to last X months, so you reset the speed limit to 45 > with a TTL on that cell version only. > > Another example is if you’re storing price for a given sku in a given region > of your retail chain. So you want to reduce the price by 20% for a 2 week > period. > Again, you set that discount to live for 2 weeks with a TTL, then revert back > to original price. > > So I guess there should be a clarification as to what is intended for the TTL > to do? > > Does that make sense? > > > > > >> On Apr 10, 2015, at 9:26 AM, Cosmin Lehene wrote: >> >> I've been initially puzzled by this, although I realize how it's likely as >> designed. >> >> >> The cell TTL expiration and compactions events can lead to either some (the >> older) data left or no data at all for a particular (row, family, >> qualifier, ts) coordinate. >> >> >> >> Write (r1, f1, q1, v1, 1) >> >> Write (r1, f1, q1, v1, 2) - TTL=1 minute >> >> >> Scenario 1: >> >> >> If a major compaction happens within a minute >> >> >> it will remove (r1, f1, q1, v1, 1) >> >> then after a minute (r1, f1, q1, v1, 2) will expire >> >> no data left >> >> >> Scenario 2: >> >> >> A minute passes >> >> (r1, f1, q1, v1, 2) expires >> >> Compaction runs.. >> >> (r1, f1, q1, v1, 1) remains >> >> >> >> This seems, by and large expected behavior, but it still seems >> "uncomfortable" that the (overall) outcome is not decided by me, but by a >> chance of event ordering. >> >> >> I wonder we'd want this to behave differently (perhaps it has been discussed >> already), but if not, it's worth a more detailed documentation in the book. >> >> >> What do you think? >> >> >> Cosmin >> >> >> >> > > The opinions expressed here are mine, while they may reflect a cognitive > thought, that is purely accidental. > Use at your own risk. > Michael Segel > michael_segel (AT) hotmail.com > > > > > > The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
Re: Nondeterministic outcome based on cell TTL and major compaction event order
The ambiguity seems to lie at the intersection of TTL and version "garbage collection" during compactions. Major compactions can lead to nondeterministic results when multiple versions are involved (slightly captured in the book http://hbase.apache.org/book.html#versions and http://www.ngdata.com/bending-time-in-hbase/ ) TTL expirations don't result in deletes (at least not in the classical sense with a tombstone). Cosmin ___ From: Michael Segel Sent: Friday, April 10, 2015 8:35 AM To: dev@hbase.apache.org Subject: Re: Nondeterministic outcome based on cell TTL and major compaction event order Interesting. There seems to be some ambiguity in what happens between a TTL and a deletion. Is the TTL a delete or is it a separate type of function? That is to say when you inserted version 2 of the cell, did you intend to just have version 2 exist for a little while and then default to version 1 or did you mean that when you inserted version 2, you wanted to delete everything prior to version 2 and then when version 2 expires, it then goes away? The documentation isn’t clear on this point. To give you an example where you wouldn’t want to have the TTL on a cell also delete prior versions… Suppose you’re storing map data in HBase. You have an attribute (speed) associated to a road link. If the road is a 65 MPH highway, then the base speed (default speed) is 65MPH. However if there’s construction planned for the road then you need to reset the speed to 45 mph while there is construction. You know that the construction is supposed to last X months, so you reset the speed limit to 45 with a TTL on that cell version only. Another example is if you’re storing price for a given sku in a given region of your retail chain. So you want to reduce the price by 20% for a 2 week period. Again, you set that discount to live for 2 weeks with a TTL, then revert back to original price. So I guess there should be a clarification as to what is intended for the TTL to do? Does that make sense? > On Apr 10, 2015, at 9:26 AM, Cosmin Lehene wrote: > > I've been initially puzzled by this, although I realize how it's likely as > designed. > > > The cell TTL expiration and compactions events can lead to either some (the > older) data left or no data at all for a particular (row, family, qualifier, > ts) coordinate. > > > > Write (r1, f1, q1, v1, 1) > > Write (r1, f1, q1, v1, 2) - TTL=1 minute > > > Scenario 1: > > > If a major compaction happens within a minute > > > it will remove (r1, f1, q1, v1, 1) > > then after a minute (r1, f1, q1, v1, 2) will expire > > no data left > > > Scenario 2: > > > A minute passes > > (r1, f1, q1, v1, 2) expires > > Compaction runs.. > > (r1, f1, q1, v1, 1) remains > > > > This seems, by and large expected behavior, but it still seems > "uncomfortable" that the (overall) outcome is not decided by me, but by a > chance of event ordering. > > > I wonder we'd want this to behave differently (perhaps it has been discussed > already), but if not, it's worth a more detailed documentation in the book. > > > What do you think? > > > Cosmin > > > > The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
