Ok… In your example the put has an earlier time T1 than T2?
I’m assuming that the orders are reversed due to something in the memcache? Also what happens to the data within the memcache that hasn’t been written to file and you insert a delete after a put() ? As to why you would want to TTL on a column that doesn’t always use a TTL? I used this example in a different post… Imagine you have a road link which has an attribute of speed. You could have construction, or variable speed limits. So you would want to change the speed limit with a TTL. Or you’re a retailer and you’re offering a 20% discount on a product for a limited time only? Sure, these are bad examples because in reality the database is a sync and the application would manage these type of issues. > On Apr 18, 2015, at 12:23 AM, lars hofhansl <la...@apache.org> wrote: > > The formatting did not come out right. Lemme try again... > > > Just came here to say that. From our (maybe not clearly enough) defined > semantics this how it should behave. > > It _is_ confusing, though, since compactions are - in a sense - just > optimizations that run in the background to prevent the number of HFiles to > be unbounded. > In this case the schedule of the compactions influences the outcome. > > Note that even tombstone markers can be confusing. Here's another confusing > example: > 1. delete (r1, f1, q1, T2) > 2. put (r1, f1, q1, v1, T1) > > If a compaction happens after #1 but before #2 the put will remain: > delete > compaction > put (remains visible) > > If the compaction happens after #2 the put will be affected by the delete and > hence removed: > delete > put > compaction (will remove the put) > > Notice though that both of these examples _are_ a bit weird. > Why would only a newer version of the cell have a TTL? > Why would you date a delete into the future? > > -- Lars > > > > > ________________________________ > From: lars hofhansl <la...@apache.org> > To: "dev@hbase.apache.org" <dev@hbase.apache.org> > Sent: Friday, April 17, 2015 10:18 PM > Subject: Re: Nondeterministic outcome based on cell TTL and major compaction > event order > > > Just came here to say that. From our (maybe not clearly enough) defined > semantics this how it should behave. > > It _is_ confusing, though, since compactions are - in a sense - just > optimizations that run in the background to prevent the number of HFiles to > be unbounded.In this case the schedule of the compactions influences the > outcome. > Note that even tombstone markers can be confusing. Here's another confusing > example:1. delete (r1, f1, q1, T2)2. put (r1, f1, q1, v1, T1) > If a compaction happens after #1 but before #2 the put will > remain:deletecompactionput (remains visible) > > If the compaction happens after #2 the put will be affected by the delete and > hence removed.deleteputcompaction (will remove the put) > > Notice though that both of these examples _are_ a bit weird.Why would only a > newer version of the cell have a TTL?Why would you date a delete into the > future? > -- Lars > > From: Sean Busbey <bus...@cloudera.com> > > > > To: dev <dev@hbase.apache.org> > Sent: Friday, April 17, 2015 4:52 PM > Subject: Re: Nondeterministic outcome based on cell TTL and major compaction > event order > > If you have max versions set to 1 (the default), then c1 should be removed > at compaction time if c2 still exists then. > > -- > Sean > > > On Apr 17, 2015 6:41 PM, "Michael Segel" <michael_se...@hotmail.com> wrote: > >> Ok, >> So then if you have a previous cell (c1) and you insert a new cell c2 that >> has a TTL of lets say 5 mins, then c1 should always exist? >> That is my understanding but from Cosmin’s post, he’s saying its >> different. And that’s why I don’t understand. You couldn’t lose the cell >> c1 at all. >> Compaction or no compaction. >> >> That’s why I’m confused. Current behavior doesn’t match the expected >> contract. >> >> -Mike >> >>> On Apr 17, 2015, at 4:37 PM, Andrew Purtell <apurt...@apache.org> wrote: >>> >>> The way TTLs work today is they define the interval of time a cell >>> exists - exactly as that. There is no tombstone laid like a normal >>> delete. Once the TTL elapses the cell just ceases to exist to normal >>> scanners. The interaction of expired cells, multiple versions, minimum >>> versions, raw scanners, etc. can be confusing. We can absolutely >>> revisit this. >>> >>> A cell with an expired TTL could be treated as the combination of >>> tombstone and the most recent value it lays over. This is not how the >>> implementation works today, but could be changed for an upcoming major >>> version like 2.0 if there's consensus to do it. >>> >>> >>>> On Apr 10, 2015, at 7:26 AM, Cosmin Lehene <cleh...@adobe.com> wrote: >>>> >>>> I've been initially puzzled by this, although I realize how it's likely >> as designed. >>>> >>>> >>>> The cell TTL expiration and compactions events can lead to either some >> (the older) data left or no data at all for a particular (row, family, >> qualifier, ts) coordinate. >>>> >>>> >>>> >>>> Write (r1, f1, q1, v1, 1) >>>> >>>> Write (r1, f1, q1, v1, 2) - TTL=1 minute >>>> >>>> >>>> Scenario 1: >>>> >>>> >>>> If a major compaction happens within a minute >>>> >>>> >>>> it will remove (r1, f1, q1, v1, 1) >>>> >>>> then after a minute (r1, f1, q1, v1, 2) will expire >>>> >>>> no data left >>>> >>>> >>>> Scenario 2: >>>> >>>> >>>> A minute passes >>>> >>>> (r1, f1, q1, v1, 2) expires >>>> >>>> Compaction runs.. >>>> >>>> (r1, f1, q1, v1, 1) remains >>>> >>>> >>>> >>>> This seems, by and large expected behavior, but it still seems >> "uncomfortable" that the (overall) outcome is not decided by me, but by a >> chance of event ordering. >>>> >>>> >>>> I wonder we'd want this to behave differently (perhaps it has been >> discussed already), but if not, it's worth a more detailed documentation in >> the book. >>>> >>>> >>>> What do you think? >>>> >>>> >>>> Cosmin >>>> >>>> >>>> >>>> >>> >>> -- >>> Best regards, >>> >>> - Andy >>> >>> Problems worthy of attack prove their worth by hitting back. - Piet >>> Hein (via Tom White) >>> >> >> The opinions expressed here are mine, while they may reflect a cognitive >> thought, that is purely accidental. >> Use at your own risk. >> Michael Segel >> michael_segel (AT) hotmail.com >> >> >> >> >> >> > The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
