Re: [VOTE] KIP-82 Add Record Headers

2017-02-14 Thread Renu Tewari
+1  after the comprehensive discussion great to see this moving to a vote.



On Tue, Feb 14, 2017 at 1:07 PM, Onur Karaman 
wrote:

> +1
>
> On Tue, Feb 14, 2017 at 10:35 AM, radai 
> wrote:
>
> > +1 from me.
> >
> > also - a more usable link to the discussion thread:
> > http://markmail.org/message/x5wlkieexinovsz3
> >
> > On Tue, Feb 14, 2017 at 9:42 AM, Michael Pearce 
> > wrote:
> >
> > > Hi all,
> > >
> > > We would like to start the voting process for KIP-82 – Add record
> > headers.
> > > The KIP can be found
> > > at
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 82+-+Add+Record+Headers
> > >
> > > Discussion thread(s) can be found here:
> > >
> > > http://search-hadoop.com/m/Kafka/uyzND1nSTOHTvj81?subj=
> > > Re+DISCUSS+KIP+82+Add+Record+Headers
> > > http://search-hadoop.com/m/Kafka/uyzND1Arxt22Tvj81?subj=
> > > Re+DISCUSS+KIP+82+Add+Record+Headers
> > > http://search-hadoop.com/?project=Kafka&q=KIP-82
> > >
> > >
> > >
> > > Thanks,
> > > Mike
> > >
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>


Re: [DISCUSS] KIP-109: Old Consumer Deprecation

2017-01-10 Thread Renu Tewari
Hi Ismael,
What are the expected timelines we are talking about between the major
releases? At LI we are expecting to have atleast 1 year between the old
consumer deprecation and removal so we have enough time to upgrade all
applications. The rollout to new consumer has hit many hurdles so hasn't
proceeded at the expected pace. What impact would an official deprecation
have on applications?  Any warnings would be disruptive and would prefer
that happens when there is a migration plan in place so we have a bound on
how long it will take. There are too many unknowns at this time.

Thoughts on timelines?

regards
Renu

On Mon, Jan 9, 2017 at 6:34 PM, Ismael Juma  wrote:

> Hi Joel,
>
> Great to hear that LinkedIn is likely to implement KAFKA-4513. :)
>
> Yes, the KIP as it stands is a compromise given the situation. We could
> push the deprecation to the subsequent release: likely to be 0.11.0.0 since
> there are a number of KIPs that require message format changes. This would
> mean that the old consumers would not be removed before 0.12.0.0 (the major
> release after 0.11.0.0). Would that work better for you all?
>
> Ismael
>
> On Tue, Jan 10, 2017 at 12:52 AM, Joel Koshy  wrote:
>
> > >
> > >
> > > The ideal scenario would be for us to provide a tool for no downtime
> > > migration as discussed in the original thread (I filed
> > > https://issues.apache.org/jira/browse/KAFKA-4513 in response to that
> > > discussion). There are a few issues, however:
> > >
> > >- There doesn't seem to be much demand for it (outside of LinkedIn,
> at
> > >least)
> > >- No-one is working on it or has indicated that they are planning to
> > >work on it
> > >- It's a non-trivial change and it requires a good amount of testing
> > to
> > >ensure it works as expected
> > >
> >
> > For LinkedIn: while there are a few consuming applications for which the
> > current shut-down and restart approach to migration will suffice, I doubt
> > we will be able to do this for majority of services that are outside our
> > direct control. Given that a seamless migration is a pre-req for us to
> > switch to the new consumer widely (there are a few use-cases already on
> it)
> > it is something that we (LinkedIn) will likely implement although we
> > haven't done further brainstorming/improvements beyond what was proposed
> in
> > the other deprecation thread.
> >
> >
> > > In the meantime, we have this suboptimal situation where the old
> > consumers
> > > are close to unmaintained even though we don't say it outright: they
> > don't
> >
> > get new features (basic things like security are missing) and bug fixes
> are
> > > rare. In practice, the old clients have been deprecated a while back,
> we
> > >
> >
> > Agreed that it is suboptimal, but the reality is that LI and I think a
> few
> > other companies are still to a large extent on the old consumer and will
> be
> > for at least a good part of this year. This does mean that we have the
> > overhead of maintaining some internal workarounds for the old consumer.
> >
> >
> > > just haven't made it official. This proposal is about rectifying that
> so
> > > that we communicate our intentions to our users more clearly. As Vahid
> > > said, this KIP is not about changing how we maintain the existing code.
> > >
> > > The KIP that proposes the removal of all the old clients will be more
> > > interesting, but it doesn't exist yet. :)
> > >
> >
> > Deprecating *after* providing a sound migration path still seems to be
> the
> > right thing to do but if there isn't any demand for it then maybe that's
> a
> > reasonable compromise. I'm still surprised that more users aren't as
> > impacted by this and as mentioned earlier, it could be an issue of
> > awareness but I'm not sure that deprecating before a migration path is in
> > place would be considered best-practice in raising awareness.
> >
> > Thanks,
> >
> > Joel
> >
> >
> >
> > >
> > > Ismael
> > >
> > > On Fri, Jan 6, 2017 at 3:27 AM, Vahid S Hashemian <
> > > vahidhashem...@us.ibm.com
> > > > wrote:
> > >
> > > > One thing that probably needs some clarification is what is implied
> by
> > > > "deprecated" in the Kafka project.
> > > > I googled it a bit and it doesn't seem that deprecation
> conventionally
> > > > implies termination of support (or anything that could negatively
> > impact
> > > > existing users). That's my interpretation too.
> > > > It would be good to know if Kafka follows a different interpretation
> of
> > > > the term.
> > > >
> > > > If my understanding of the term is correct, since we are not yet
> > > targeting
> > > > a certain major release in which the old consumer will be removed, I
> > > don't
> > > > see any harm in marking it as deprecated.
> > > > There will be enough time to plan and implement the migration, if the
> > > > community decides that's the way to go, before phasing it out.
> > > >
> > > > At the minimum new Kafka users will pick the Java consumer without
> any
> > > > confu

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

2016-11-14 Thread Renu Tewari
Is upping the magic byte to 2 needed?

In your example say
For broker api version at or above 0.10.2 the tombstone bit will be used
for log compaction deletion.
If the producerequest version is less than 0.10.2 and the message is null,
the broker will up convert to set the tombstone bit on
If the producerequest version is at or above 0.10.2 then the tombstone bit
value is unchanged
Either way the new version of broker only  uses the tombstone bit
internally.

thanks
Renu

On Mon, Nov 14, 2016 at 8:31 PM, Becket Qin  wrote:

> If we follow the current way of doing this format change, it would work the
> following way:
>
> 0. Bump up the magic byte to 2 to indicate the tombstone bit is used.
>
> 1. On receiving a ProduceRequest, broker always convert the messages to the
> configured message.format.version.
> 1.1 If the message version does not match the configured
> message.format.version, the broker will either up convert or down convert
> the message. In that case, users pay the performance cost of re-compression
> if needed.
> 1.2 If the message version matches the configured message.format.version,
> the broker will not do the format conversion and user may save the
> re-compression cost if the message.format.version is on or above 0.10.0.
>
> 2. On receiving a FetchRequest, the broker check the FetchRequest version
> to see if the consumer supports the configured message.format.version or
> not. If the consumer does not support it, down conversion is required and
> zero copy is lost. Otherwise zero copy is used to return the FetchResponse.
>
> Notice that setting message.format.version to 0.10.2 is equivalent to
> always up convert if needed, but that also means to always down convert if
> there is an old consumer.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Mon, Nov 14, 2016 at 1:43 PM, Michael Pearce 
> wrote:
>
> > I like the idea of up converting and then just having the logic to look
> > for tombstones. It makes that quite clean in nature.
> >
> > It's quite late here in the UK, so I fully understand / confirm I
> > understand what you propose could you write it on the kip wiki or fully
> > describe exactly how you see it working, so uk morning I could read
> through?
> >
> > Thanks all for the input on this it is appreciated.
> >
> >
> > Sent using OWA for iPhone
> > 
> > From: Mayuresh Gharat 
> > Sent: Monday, November 14, 2016 9:28:16 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> >
> > Hi Michael,
> >
> > Just another thing that came up during my discussion with Renu and I
> wanted
> > to share it.
> >
> > Other thing we can do to handle a mixture of old and new clients is when
> > once the new broker with this KIP is deployed, the new code should check
> > the request version from older producer we can up convert it with a
> > tombstone marker when appending the message to the log. This is similar
> to
> > down converting messages for older clients.
> >
> > If this is possible then the broker in this case has to rely only on the
> > tombstone marker for log compaction. Using this approach we preserve the
> > description of when to update the magic byte as described here :
> > https://kafka.apache.org/documentation#messageformat (1 byte "magic"
> > identifier to allow format changes).
> >
> > In stage 2, if we don't want open source kafka to make the decision of
> > deprecation of null value for log compaction (which is the approach I
> would
> > prefer as an end state) since there are some concerns on this, individual
> > companies/orgs can then have a horizontal initiative at their convenience
> > to move to stage 2 by asking all app users to upgrade to new kafka
> clients.
> >
> > Thanks,
> >
> > Mayuresh
> >
> > On Mon, Nov 14, 2016 at 11:50 AM, Becket Qin 
> wrote:
> >
> > > Michael,
> > >
> > > Yes, I am OK with stage 1. We can discuss about stage 2 later but this
> > > sounds really an organization specific decision to deprecate an API. It
> > > does not seem a general need in open source Kafka to only support
> > tombstone
> > > bit , which is a bad thing for people who are still running old clients
> > in
> > > the community. This is exactly why we want to have magic byte bump -
> > there
> > > should be only one definitive way to interpret a message with a given
> > magic
> > > byte. We should avoid re-define the interpretation of a message with an
> > > existing magic byte. The interpretation of an old message format should
> > not
> > > be changed and should always be supported until deprecated.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Nov 14, 2016 at 11:28 AM, Mayuresh Gharat <
> > > gharatmayures...@gmail.com> wrote:
> > >
> > > > I am not sure about "If I understand correctly, you want to let the
> > > broker
> > > > to reject requests
> > > > from old clients to ensure everyone in an organization has upgraded,
> > > > right?"
> > > >
> > > > I don't

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

2016-10-31 Thread Renu Tewari
Congratulations Becket!! Absolutely thrilled to hear this. Well deserved!

regards
renu


On Mon, Oct 31, 2016 at 10:35 AM, Joel Koshy  wrote:

> The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a
> committer and we are pleased to announce that he has accepted!
>
> Becket has made significant contributions to Kafka over the last two years.
> He has been deeply involved in a broad range of KIP discussions and has
> contributed several major features to the project. He recently completed
> the implementation of a series of improvements (KIP-31, KIP-32, KIP-33) to
> Kafka’s message format that address a number of long-standing issues such
> as avoiding server-side re-compression, better accuracy for time-based log
> retention, log roll and time-based indexing of messages.
>
> Congratulations Becket! Thank you for your many contributions. We are
> excited to have you on board as a committer and look forward to your
> continued participation!
>
> Joel
>


Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

2016-10-26 Thread Renu Tewari
+1 @Magnus.
It is also in line with traditional use of the magic field to indicate a
change in the format of the message. Thus a change in the magic field
indicates a different "schema" which in this case would reflect adding a
new field, removing a field or changing the type of fields etc.

The version number bump does not change the message format but just the
interpretation of the values.

Going with what @Michael suggested on keeping it simple we don't need to a
add a new field to indicate a tombstone and therefore require a change in
the magic byte field. The attribute bit is sufficient for indicating the
new interpretation of attribute.bit5 to indicate a tombstone.

Bumping the version number and changing the attribute bit keeps it simple.


Thanks
Renu



On Wed, Oct 26, 2016 at 10:09 AM, Mayuresh Gharat <
gharatmayures...@gmail.com> wrote:

> I see the reasoning Magnus described.
> If you check the docs https://kafka.apache.org/documentation#messageformat
> ,
> it says :
>
> 1 byte "magic" identifier to allow format changes, value is 0 or 1
>
> Moreover as per comments in the code :
> /**
>* The "magic" value
>* When magic value is 0, the message uses absolute offset and does not
> have a timestamp field.
>* When magic value is 1, the message uses relative offset and has a
> timestamp field.
>*/
>
> Since timeStamp was added as an actual field we bumped the the magic byte
> to 1 for this change.
> But since we are not adding an actual field, we can do away with bumping up
> the magic byte.
>
> If we really want to go the standard route of bumping up the magic byte for
> any change to message format we should actually add a new field for
> handling log compaction instead of just using the attribute field, which
> might sound like an overkill.
>
> Thanks,
>
> Mayuresh
>
>
>
>
>
>
> On Wed, Oct 26, 2016 at 1:32 AM, Magnus Edenhill 
> wrote:
>
> > 2016-10-25 21:36 GMT+02:00 Nacho Solis :
> >
> > > I think you probably require a MagicByte bump if you expect correct
> > > behavior of the system as a whole.
> > >
> > > From a client perspective you want to make sure that when you deliver a
> > > message that the broker supports the feature you're expecting
> > > (compaction).  So, depending on the behavior of the broker on
> > encountering
> > > a previously undefined bit flag I would suggest making some change to
> > make
> > > certain that flag-based compaction is supported.  I'm going to guess
> that
> > > the MagicByte would do this.
> > >
> >
> > I dont believe this is needed since it is already attributed through the
> > request's API version.
> >
> > Producer:
> >  * if a client sends ProduceRequest V4 then attributes.bit5 indicates a
> > tombstone
> >  * if a clients sends ProduceRequest  > and value==null indicates a tombstone
> >  * in both cases the on-disk messages are stored with attributes.bit5 (I
> > assume?)
> >
> > Consumer:
> >  * if a clients sends FetchRequest V4 messages are sendfile():ed directly
> > from disk (with attributes.bit5)
> >  * if a client sends FetchRequest  > translated from attributes.bit5 to value=null as required.
> >
> >
> > That's my understanding anyway, please correct me if I'm wrong.
> >
> > /Magnus
> >
> >
> >
> > > On Tue, Oct 25, 2016 at 10:17 AM, Magnus Edenhill 
> > > wrote:
> > >
> > > > It is safe to assume that a previously undefined attributes bit will
> be
> > > > unset in protocol requests from existing clients, if not, such a
> client
> > > is
> > > > already violating the protocol and needs to be fixed.
> > > >
> > > > So I dont see a need for a MagicByte bump, both broker and client has
> > the
> > > > information it needs to construct or parse the message according to
> > > request
> > > > version.
> > > >
> > > >
> > > > 2016-10-25 18:48 GMT+02:00 Michael Pearce :
> > > >
> > > > > Hi Magnus,
> > > > >
> > > > > I was wondering if I even needed to change those also, as
> technically
> > > > > we’re just making use of a non used attribute bit, but im not 100%
> > that
> > > > it
> > > > > be always false currently.
> > > > >
> > > > > If someone can say 100% it will already be set false with current
> and
> > > > > historic bit wise masking techniques used over the time, we could
> do
> > > away
> > > > > with both, and simply just start to use it. Unfortunately I don’t
> > have
> > > > that
> > > > > historic knowledge so was hoping it would be flagged up in this
> > > > discussion
> > > > > thread ☺
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > > On 10/25/16, 5:36 PM, "Magnus Edenhill" 
> wrote:
> > > > >
> > > > > Hi Michael,
> > > > >
> > > > > With the version bumps for Produce and Fetch requests, do you
> > > really
> > > > > need
> > > > > to bump MagicByte too?
> > > > >
> > > > > Regards,
> > > > > Magnus
> > > > >
> > > > >
> > > > > 2016-10-25 18:09 GMT+02:00 Michael Pearce <
> michael.pea...@ig.com
> > >:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I would like

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

2016-10-10 Thread Renu Tewari
Hi David
  This is a very timely KIP given the number of use cases in the streams
processing pipeline than need consumed log retention management.

Some questions that Becket and Dong asked just wanted to make sure are
described in the KIP.

1. How is the configuration setup per topic to know what is the set of
consumer groups that are "subscribed" to this topic whose committed offsets
will be tracked. Can we have more details on how this will be dynamically
tracked as consumers come and go.
2. Is there a timeout to determine if a consumer group has stopped
committing offsets to topic partitions that they had earlier consumed? Or
the consumed log retention will track each known consumer/consumers groups
committed offset and stop any cleaning if a consumer disappears after
consuming. This is to Dong's earlier question.
3. Can the log.retention value be set to 0 to indicate the log is set to be
cleaned to the min committed offset immediately after it has been consumed?

4. What guarantee is given on when the consumed log will eventually be
cleaned. If the log.retention timeout is enabled for a consumed offset and
a new consumer starts consuming from the beginning then is the min
committed offset value changed and the timer based on log.retention timeout
restarted?

 This kind of all relates to active and inactive consumers and if the set
changes dynamically how does the consumed log retention actually make
progress.

regards
renu


On Mon, Oct 10, 2016 at 1:05 AM, Dong Lin  wrote:

> Hey David,
>
> Thanks for reply. Please see comment inline.
>
> On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L) 
> wrote:
>
> > Hi Dong
> >Thanks for the questions:
> >
> > 1.  Now we don't distinguish inactive or active groups. Because in some
> > case maybe inactive group will become active again, and using the
> previous
> > commit offset.
> >
> > So we will not delete the log segment in the consumer retention if there
> > are some groups consume but not commit, but the log segment can be
> delete by
> >  the force retention.
> >
>
> So in the example I provided, the consumed log retention will be
> effectively disabled, right? This seems to be a real problem in operation
> -- we don't want log retention to be un-intentionally disabled simply
> because someone start a tool to consume from that topic. Either this KIP
> should provide a way to handle this, or there should be a way for operator
> to be aware of such case and be able to re-eanble consumed log retention
> for the topic. What do you think?
>
>
>
> > 2.  These configs are used to determine the out of date time of the
> > consumed retention, like the parameters of the force retention
> > (log.retention.hours, log.retention.minutes, log.retention.ms). For
> > example, users want the save the log for 3 days, after 3 days, kafka will
> > delete the log segments which are
> >
> > consumed by all the consumer group.  The log retention thread need these
> > parameters.
> >
> > It makes sense to have configs such as log.retention.ms -- it is used to
> make data available for up to a configured amount of time before it is
> deleted. My question is what is the use-case for making log available for
> another e.g. 3 days after it has been consumed by all consumer groups. The
> purpose of this KIP is to allow log to be deleted right as long as all
> interested consumer groups have consumed it. Can you provide a use-case for
> keeping log available for longer time after it has been consumed by all
> groups?
>
>
> >
> > Thanks,
> > David
> >
> >
> > > Hey David,
> > >
> > > Thanks for the KIP. Can you help with the following two questions:
> > >
> > > 1) If someone start a consumer (e.g. kafka-console-consumer) to
> consume a
> > > topic for debug/validation purpose, a randome consumer group may be
> > created
> > > and offset may be committed for this consumer group. If no offset
> commit
> > is
> > > made for this consumer group in the future, will this effectively
> > > disable consumed log retention for this topic? In other words, how do
> > this
> > > KIP distinguish active consumer group from inactive ones?
> > >
> > > 2) Why do we need new configs such as log.retention.commitoffset.
> hours?
> > Can
> > >we simply delete log segments if consumed log retention is enabled for
> > this
> > > topic and all consumer groups have consumed messages in the log
> segment?
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > >
> > >On Sat, Oct 8, 2016 at 2:15 AM, Pengwei (L) 
> > wrote:
> > >
> > > > Hi Becket,
> > > >
> > > >   Thanks for the feedback:
> > > > 1.  We use the simple consumer api to query the commit offset, so we
> > don't
> > > > need to specify the consumer group.
> > > > 2.  Every broker using the simple consumer api(OffsetFetchKey) to
> query
> > > > the commit offset in the log retention process.  The client can
> commit
> > > > offset or not.
> > > > 3.  It does not need to distinguish the follower brokers or leader
> > > > brokers,  every brokers can query.
> > 

Re: [DISCUSS] KIP-82 - Add Record Headers

2016-10-06 Thread Renu Tewari
+1 Nacho, Radai, Mayuresh
1) yes for unordered keys. Given the expected number of headers is not that
high linear searching is ok. The overhead of ordering and the dependence on
client implementations makes it error prone.
2) yes on proposed key space. Adding structure makes it easier to manage
and delegate handling for different use cases. Also simplifies debugging
and special casing.

Thanks
Renu

On Thu, Oct 6, 2016 at 3:31 PM, Mayuresh Gharat 
wrote:

> +1 Nacho, Radai.
>
> Ordering the Keys would help if we were gonna look at the headers linearly
> but given the disadvantage that the client implementations have to know the
> order of headers in order that the reading system in the pipeline doesn't
> break, unordered list sounds better.
>
> Thanks,
>
> Mayuresh
>
>
> On Thu, Oct 6, 2016 at 2:46 PM, Nacho Solis 
> wrote:
>
> > I'm also
> >
> > 1. no  (ordered keys)
> > 2. yes (propose key space)
> >
> >
> > 1. I don't think there is going to be much savings in ordering the keys.
> > I'm assuming some parsing will happen either way. Ordering the keys would
> > be useful if we were doing linear search on the headers, and even then,
> the
> > performance difference would be small for any reasonable number of
> headers
> > (even anything that fits in 1 meg).
> >
> > However, I think that it's likely that whoever is looking at the headers
> is
> > going to want to search for plugins for any header in existence, as such,
> > it's going to have to iterate over the whole header set. So, for every
> > header, look up the plugin, not the other way around.  Even if we did it
> > the other way around (for every plugin, search if there is a header
> > present) we would expect only to have an algorithm that is O(n) and only
> > iterate once over the list. We wouldn't need to iterate more than once.
> >
> > Given this, the code overhead of ordering the headers when something is
> > inserted and such is a bigger pain than dealing with a potentially
> > unordered list.
> >
> >
> > 2. I like structure and to reduce the play space for potential keys.
> This
> > will allow us to do filter and know when we're testing. At the same time,
> > we're reserving a lot of space for future usages. However, if there is no
> > agreement on this I don't think it would be a blocker.  I just want to
> make
> > sure we have some order and if possible contiguous ranges for similar
> > usages.
> >
> > Nacho
> >
> >
> > On Thu, Oct 6, 2016 at 2:26 PM, radai 
> wrote:
> >
> > > 1. tending towards no, but I dont have any strong opinions on header
> > > ordering. it offers a potential speedup for header lookup in a
> serialized
> > > blob (in wire format) but that goes away if the headers are fully
> > > serialized/deserialized always. on the downside its an implementation
> > > detail that 3rd party impls would need to worry about, and would be
> hard
> > to
> > > diagnose if they fail to. its also less friendly to high performance io
> > > (think about appending headers to an existing blob in pass-through
> > > components like mirror-maker vs write to the middle) - its still
> possible
> > > though. however, the kafka code base is far from being iovec friendly
> > > anyway.
> > >
> > > 2. yes.
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Oct 6, 2016 at 8:58 AM, K Burstev 
> wrote:
> > >
> > > > @Mayuresh
> > > >
> > > > Yes exactly, it is a real nasty race issue.
> > > >
> > > > This is why I look forward to being able to trash our custom
> workaround
> > > :)
> > > >
> > > > Kostya
> > > >
> > > >
> > > > 06.10.2016, 02:36, "Mayuresh Gharat" :
> > > > > @Kostya
> > > > >
> > > > > Regarding "To get around this we have an awful *cough* solution
> > whereby
> > > > we
> > > > > have to send our message wrapper with the headers and null content,
> > and
> > > > > then we have an application that has to consume from all the
> > compacted
> > > > > topics and when it sees this message it produces back in a null
> > payload
> > > > > record to make the broker compact it out."
> > > > >
> > > > >  ---> This has a race condition, right?
> > > > >
> > > > > Suppose the producer produces a message with headers and null
> content
> > > at
> > > > > time To to Kafka.
> > > > >
> > > > > Then the producer, at time To + 1, sends another message with
> headers
> > > and
> > > > > actual content to Kafka.
> > > > >
> > > > > What we expect is that the application that is consuming and then
> > > > producing
> > > > > same message with null payload should happen at time To + 0.5, so
> > that
> > > > the
> > > > > message at To + 1 is not deleted.
> > > > >
> > > > > But there is no guarantee here.
> > > > >
> > > > > If the null payload goes in to Kafka at time To + 2, then
> essentially
> > > you
> > > > > loose the second message produced by the producer at time To + 1.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Mayuresh
> > > > >
> > > > > On Wed, Oct 5, 2016 at 6:13 PM, Joel Koshy 
> > > wrote:
> > > > >
> > > > >>  @Nacho
> > > > >>
> > > > >>  > > 

Status of KIP-68: Add a consumed log retention

2016-09-20 Thread Renu Tewari
Hi All,
  Wanted to find out more on the status of KIP-68. There are no details in
the discussion thread and the pointers are not updated. Is this being
actively worked on? This is clearly needed to support smarter and more cost
effective retentions.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-68+Add+a+consumed+log+retention+before+log+retention

best
Renu


Re: Kafka KIP meeting Sep 13 at 11:00am PST

2016-09-12 Thread Renu Tewari
Hi Ismael
Could you please add me to the invite.

rtew...@linkedin.com

regards
Renu


On Mon, Sep 12, 2016 at 8:39 AM, Ismael Juma  wrote:

> Hi, Everyone,
>
> We plan to have a Kafka KIP meeting this coming Tuesday at 11:00am PST. If
> you plan to attend but haven't received an invite, please let me know. The
> following is the tentative agenda.
>
> Agenda:
> KIP-54: Sticky Partition Assignment Strategy
> KIP-72: Allow Sizing Incoming Request Queue in Bytes
>
> Thanks,
> Ismael
>