Naively (I may be missing something) it seems much easier to flush a single memtable to more than one stable on disk (static and non static) and then allow for separate compaction of those
> On May 1, 2015, at 9:06 AM, Benedict Elliott Smith > <belliottsm...@datastax.com> wrote: > > It also doesn't solve the atomicity problem, which is its own challenge. We > would probably need to merge the memtables for the entire keyspace/node, > and split them out into their own sstables on flush. Or introduce mutual > exclusion at the partition key level for the node. > > On Fri, May 1, 2015 at 3:01 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > >> I'm down for adding JOIN support within a partition, eventually. I can see >> a lot of stuff I'd rather prioritize higher in the short term though. >> >> On Fri, May 1, 2015 at 8:44 AM, Jonathan Haddad <j...@jonhaddad.com> wrote: >> >>> I think what Benedict has described feels very much like a very >> specialized >>> version of the following: >>> >>> 1. Updates to different tables in a batch become atomic if the node is a >>> replica for the partition >>> 2. Supporting Inner joins if the partition key is the same in both >> tables. >>> >>> I'd rather see join support personally :) >>> >>> Jon >>> >>> On Fri, May 1, 2015 at 6:38 AM graham sanderson <gra...@vast.com> wrote: >>> >>>> I 100% agree with Benedict, but just to be clear about my use case >>>> >>>> 1) We have state of lets say real estate listings >>>> 2) We get field level deltas for them >>>> 3) Previously we would store the base state all the deltas in partition >>>> and roll them up from the beginning of time (this was a prototype and >>> silly >>>> since there was no expiration strategy) >>>> 4) Preferred plan is to keep current state in a static map (i.e. one >>> delta >>>> field only updates one cell) - we are MVCC but in the common case the >>>> latest version will be what we want >>>> 5) However we require history, so we’d use the partition to keep TTL >>>> deltas going backwards from the now state - this seems like a common >>>> pattern people would want. Note also that sometimes we might need to >>> apply >>>> reverse deltas if C* is ahead of our SOLR indexes >>>> >>>> The static columns and the regular columns ARE completely different in >>>> behavior/lifecycle, so I’d definitely vote for them being treated as >>> such. >>>> >>>> >>>>> On May 1, 2015, at 7:27 AM, Benedict Elliott Smith < >>>> belliottsm...@datastax.com> wrote: >>>>> >>>>>> >>>>>> How would it be different from creating an actual real extra table >>>> instead? >>>>> >>>>> >>>>> There's nothing that warrants making the codebase more complex to >>>>>> accomplish something it already does. >>>>> >>>>> >>>>> As far as I was aware, the only point of static columns was to >> support >>>> the >>>>> thrift ability to mutate and read them in the same expression, with >>>>> atomicity and isolation. As to whether or not it is more complex, I'm >>> not >>>>> at all convinced that it would be. We have had a lot of unexpected >>>> special >>>>> casing added to ensure they behave correctly (e.g. paging is broken), >>> and >>>>> have complicated the comparison/slice logic to accommodate them, so >>> that >>>> it >>>>> is harder to reason about (and to optimise). They also have very >>>> different >>>>> compaction characteristics, so the complexity on the user is >> increased >>>>> without their necessarily realising it. All told, it introduces a lot >>>> more >>>>> subtlety of behaviour than there would be with a separate set of >>>> sstables, >>>>> or perhaps a separate file attached to each sstable. >>>>> >>>>> Of course, we've already implemented it as a specialisation of the >>>>> slice/comparator, I think because it seemed like the least frictional >>>> path >>>>> to do so, but that doesn't mean it is the least complex. It does mean >>>> it's >>>>> the least work (assuming we're now on top of the bugs), which is its >>> own >>>>> virtue. >>>>> >>>>> There are some advantages to having them managed separately, and >>>> advantages >>>>> to having them combined. Combined, for small partitions, they can be >>> read >>>>> in the same seek. However for large partitions this is no longer >> true, >>>> and >>>>> we may behave much worse by polluting the page cache with lots of >>>> unwanted >>>>> data that is adjacent to the static columns. If they were managed >>>>> separately, the page cache would be populated mostly with other >> static >>>>> columns, which may be more likely of use. We could quite easily have >> a >>>>> "static column" cache, also, and completely avoid merging them. Or at >>>> least >>>>> we could easily read them with collectTimeOrderedData instead of >>>>> collectAllData semantics. >>>>> >>>>> All told, it certainly isn't a terrible idea, and shouldn't be >>> dismissed >>>> so >>>>> readily. Personally I think in the long run whether or not we manage >>>> static >>>>> columns together with non-static columns is dependent on if we intend >>> to >>>>> add tiered "static" columns (i.e., if each level of clustering >>> component >>>>> can have columns associated with it). If we do, we should definitely >>> keep >>>>> it all inline. If not, it probably permits a lot better behaviour to >>>>> separate them, since it's easier to reason about and improve their >>>> distinct >>>>> characteristics. >>>>> >>>>> >>>>> On Fri, May 1, 2015 at 1:24 AM, graham sanderson <gra...@vast.com> >>>> wrote: >>>>> >>>>>> Well you lose the atomicity and isolation, but in this case that is >>>>>> probably fine >>>>>> >>>>>> That said, in every interaction I’ve had with static columns, they >>> seem >>>> to >>>>>> be an odd duck (e.g. adding or complicating range slices), perhaps >>>> worthy >>>>>> of their own code path and sstables. Just food for thought. >>>>>> >>>>>>> On Apr 30, 2015, at 7:13 PM, Jonathan Haddad <j...@jonhaddad.com> >>>> wrote: >>>>>>> >>>>>>> If you want it in a separate sstable, just use a separate table. >>>> There's >>>>>>> nothing that warrants making the codebase more complex to >> accomplish >>>>>>> something it already does. >>>>>>> >>>>>>> On Thu, Apr 30, 2015 at 5:07 PM graham sanderson <gra...@vast.com> >>>>>> wrote: >>>>>>> >>>>>>>> Anyone here have an opinion; how realistic would it be to have a >>>>>> separate >>>>>>>> memtable/sstable for static columns? >>>>>>>> >>>>>>>> Begin forwarded message: >>>>>>>> >>>>>>>> *From: *Jonathan Haddad <j...@jonhaddad.com> >>>>>>>> *Subject: **Re: DateTieredCompactionStrategy and static columns* >>>>>>>> *Date: *April 30, 2015 at 3:55:46 PM CDT >>>>>>>> *To: *u...@cassandra.apache.org >>>>>>>> *Reply-To: *u...@cassandra.apache.org >>>>>>>> >>>>>>>> >>>>>>>> I suspect this will kill the benefit of DTCS, but haven't tested >> it >>> to >>>>>> be >>>>>>>> 100% here. >>>>>>>> >>>>>>>> The benefit of DTCS is that sstables are selected for compaction >>> based >>>>>> on >>>>>>>> the age of the data, not their size. When you mix TTL'ed data and >>> non >>>>>>>> TTL'ed data, you end up screwing with the "drop the entire >> SSTable" >>>>>>>> optimization. I don't believe this is any different just because >>>> you're >>>>>>>> mixing in static columns. What I think will happen is you'll end >> up >>>>>> with >>>>>>>> an sstable that's almost entirely TTL'ed with a few static columns >>>> that >>>>>>>> will never get compacted or dropped. Pretty much the worst >>> scenario I >>>>>> can >>>>>>>> think of. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Apr 30, 2015 at 11:21 AM graham sanderson < >> gra...@vast.com> >>>>>> wrote: >>>>>>>> >>>>>>>>> I have a potential use case I haven’t had a chance to prototype >>> yet, >>>>>>>>> which would normally be a good candidate for DTCS (i.e. data >>>> delivered >>>>>> in >>>>>>>>> order and a fixed TTL), however with every write we’d also be >>>> updating >>>>>> some >>>>>>>>> static cells (namely a few key/values in a static map<text.text> >>> CQL >>>>>>>>> column). There could also be explicit deletes of keys in the >> static >>>>>> map, >>>>>>>>> though that’s not 100% necessary. >>>>>>>>> >>>>>>>>> Since those columns don’t have TTL, without reading thru the code >>>> code >>>>>>>>> and/or trying it, I have no idea what effect this has on DTCS >>>> (perhaps >>>>>> it >>>>>>>>> needs to use separate sstables for static columns). Has anyone >>> tried >>>>>> this. >>>>>>>>> If not I eventually will and will report back. >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>>> >>> >> >> >> >> -- >> Jonathan Ellis >> Project Chair, Apache Cassandra >> co-founder, http://www.datastax.com >> @spyced >>
smime.p7s
Description: S/MIME cryptographic signature
