My two cents on the magic value bumping. Although the broker can infer from the request version to determine whether it should read the new attribute bit or not, currently the request version is not propagated to the replica manager. Bumping up the magic value does not seem a big overhead in this case. It also help avoid the potential ambiguity. maybe it is better to bump up the magic value.
Thanks, Jiangjie (Becket) Qin On Mon, Oct 31, 2016 at 4:17 PM, Jay Kreps <j...@confluent.io> wrote: > Xavier, > > Yeah I think that post KIP-58 it is possible to depend on the delivery of > messages in compacted topics, if you override the default compaction time. > Prior to that it was true that you could control the delete retention, but > any message including the tombstone could be compacted away prior to > delivery. That was what I meant by non-deterministic. Now that we have that > KIP I agree that at least it can be given an SLA like any other message > (which was what I was meaning by deterministic). > > -Jay > > On Fri, Oct 28, 2016 at 10:15 AM, Xavier Léauté <xav...@confluent.io> > wrote: > > > > > > > I kind of agree with James that it is a bit questionable how valuable > any > > > data in a delete marker can be since it will be deleted somewhat > > > nondeterministically. > > > > > > > One could argue that even in normal topics, assuming a time-based log > > retention policy is in place, any message will be deleted somewhat > > nondeterministally, so why treat the compacted ones any differently? To > me > > at least, the retention setting for delete messages seems to be the > > counterpart to the time-based retention setting for normal topics. > > > > Currently the semantics of the messages are in the eye of the > beholder--you > > > can choose to interpret a stream as either being appends or revisions > as > > > you choose. This proposal is changing that so that the semantics are > > > determined by the sender. > > > > > > Let's imagine someone wanted to augment this stream to include audit logs > > for each record update, e.g. which user made the change. One would want > to > > include that information as part of the message, and have the ability to > > mark a deletion. > > > > I don't think it changes the semantics in this case, you can still choose > > to interpret the data as a stream of audit log entries (inserts), > ignoring > > the tombstone flag, or you can interpret it as a table modeling only the > > latest version of each record. Whether a compacted or normal topic is > used > > shouldn't matter to the sender. > > >
