Thanks for the update. Good to know that TWCS give you more stability On Wed, Feb 8, 2017 at 6:20 PM, John Sanda <john.sa...@gmail.com> wrote:
> I wanted to provide a quick update. I was able to patch one of the > environments that is hitting the tombstone problem. It has been running > TWCS for five days now, and things are stable so far. I also had a patch to > the application code to implement date partitioning ready to go, but I > wanted to see how things went with only making the compaction changes. > > On Sun, Jan 29, 2017 at 4:05 PM, DuyHai Doan <doanduy...@gmail.com> wrote: > >> In theory, you're right and Cassandra should possibly skip reading cells >> having time < 50. But it's all theory, in practice Cassandra read chunks of >> xxx kilobytes worth of data (don't remember the exact value of xxx, maybe >> 64k or far less) so you may end up reading tombstones. >> >> On Sun, Jan 29, 2017 at 9:24 PM, John Sanda <john.sa...@gmail.com> wrote: >> >>> Thanks for the clarification. Let's say I have a partition in an SSTable >>> where the values of time range from 100 to 10 and everything < 50 is >>> expired. If I do a query with time < 100 and time >= 50, are there >>> scenarios in which Cassandra will have to read cells where time < 50? In >>> particular I am wondering if compression might have any affect. >>> >>> On Sun, Jan 29, 2017 at 3:01 PM DuyHai Doan <doanduy...@gmail.com> >>> wrote: >>> >>>> "Should the data be sorted by my time column regardless of the >>>> compaction strategy" --> It does >>>> >>>> What I mean is that an old "chunk" of expired data in SSTABLE-12 may be >>>> compacted together with a new chunk of SSTABLE-2 containing fresh data so >>>> in the new resulting SSTable will contain tombstones AND fresh data inside >>>> the same partition, but of course sorted by clustering column "time". >>>> >>>> On Sun, Jan 29, 2017 at 8:55 PM, John Sanda <john.sa...@gmail.com> >>>> wrote: >>>> >>>> Since STCS does not sort data based on timestamp, your wide partition >>>> may span over multiple SSTables and inside each SSTable, old data (+ >>>> tombstones) may sit on the same partition as newer data. >>>> >>>> >>>> Should the data be sorted by my time column regardless of the >>>> compaction strategy? I didn't think that the column timestamp came into >>>> play with respect to sorting. I have been able to review some SSTables with >>>> sstablemetadata and I can see that old/expired data is definitely living >>>> with live data. >>>> >>>> >>>> On Sun, Jan 29, 2017 at 2:38 PM, DuyHai Doan <doanduy...@gmail.com> >>>> wrote: >>>> >>>> Ok so give it a try with TWCS. Since STCS does not sort data based on >>>> timestamp, your wide partition may span over multiple SSTables and inside >>>> each SSTable, old data (+ tombstones) may sit on the same partition as >>>> newer data. >>>> >>>> When reading by slice, even if you request for fresh data, Cassandra >>>> has to scan over a lot tombstones to fetch the correct range of data thus >>>> your issue >>>> >>>> On Sun, Jan 29, 2017 at 8:19 PM, John Sanda <john.sa...@gmail.com> >>>> wrote: >>>> >>>> It was with STCS. It was on a 2.x version before TWCS was available. >>>> >>>> On Sun, Jan 29, 2017 at 10:58 AM DuyHai Doan <doanduy...@gmail.com> >>>> wrote: >>>> >>>> Did you get this Overwhelming tombstonne behavior with STCS or with >>>> TWCS ? >>>> >>>> If you're using DTCS, beware of its weird behavior and tricky >>>> configuration. >>>> >>>> On Sun, Jan 29, 2017 at 3:52 PM, John Sanda <john.sa...@gmail.com> >>>> wrote: >>>> >>>> Your partitioning key is text. If you have multiple entries per id you >>>> are likely hitting older cells that have expired. Descending only affects >>>> how the data is stored on disk, if you have to read the whole partition to >>>> find whichever time you are querying for you could potentially hit >>>> tombstones in other SSTables that contain the same "id". As mentioned >>>> previously, you need to add a time bucket to your partitioning key and >>>> definitely use DTCS/TWCS. >>>> >>>> >>>> As I mentioned previously, the UI only queries recent data, e.g., the >>>> past hour, past two hours, past day, past week. The UI does not query for >>>> anything older than the TTL which is 7 days. My understanding and >>>> expectation was that Cassandra would only scan live cells. The UI is a >>>> separate application that I do not maintain, so I am not 100% certain about >>>> the queries. I have been told that it does not query for anything older >>>> than 7 days. >>>> >>>> On Sun, Jan 29, 2017 at 4:14 AM, kurt greaves <k...@instaclustr.com> >>>> wrote: >>>> >>>> >>>> Your partitioning key is text. If you have multiple entries per id you >>>> are likely hitting older cells that have expired. Descending only affects >>>> how the data is stored on disk, if you have to read the whole partition to >>>> find whichever time you are querying for you could potentially hit >>>> tombstones in other SSTables that contain the same "id". As mentioned >>>> previously, you need to add a time bucket to your partitioning key and >>>> definitely use DTCS/TWCS. >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> - John >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> - John >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >> > > > -- > > - John >
