Re: [VOTE] KIP-354 Time-based log compaction policy

; wrote: > > > > > I don't have a strong opinion. But I think we should probably be > > > consistent with how segment.ms works, and just use 0. > > > > > > best, > > > Colin > > > > > > > > > On Wed, Sep 5, 20

[jira] [Resolved] (KAFKA-7137) ability to trigger compaction for tombstoning and GDPR

[ https://issues.apache.org/jira/browse/KAFKA-7137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brett Rann resolved KAFKA-7137. --- Resolution: Duplicate > ability to trigger compaction for tombstoning and G

Re: [VOTE] KIP-354 Time-based log compaction policy

tation will > treat it as disabled. > > It is a little bit weird to treat max.compaction.lag=0 the same as > max.compaction.lag=1. > > There might be a reason why we set the minimum of segment.ms to 1, and I > don't want to break this assumption. > > > > Xiongqi (We

Re: [VOTE] KIP-354 Time-based log compaction policy

be half of segment.ms. > We cannot provide "0" as max compaction lag. > > Here are two options. > Option 1: > Keep 0 as disabled > Option 2: > -1 (disabled), 0 (max compaction lag = segment.ms), and others. > > > > Xiongqi (Wesley) Wu > > >

Re: [VOTE] KIP-354 Time-based log compaction policy

ntion of 0 meaning those things. There are some NULLs but it seems convetion there is NULL is used where there's another setting in the hierarchy that would be used instead. On Thu, Sep 6, 2018 at 10:42 AM Brett Rann wrote: > If segment.ms can't be set to 0, then we're not being consistent

Re: [VOTE] KIP-354 Time-based log compaction policy

t.ms" to 1, we cannot set " > segment.ms" to 0. > > I also updated the title of this KIP. > > Xiongqi (Wesley) Wu > > > On Wed, Sep 5, 2018 at 4:34 PM Brett Rann > wrote: > > > I withdraw my comments on -1 since i'm in the minority. :) Can we &g

Re: [VOTE] KIP-354 Time-based log compaction policy

n" is a weaker guarantee than setting any value less > than 24 hours. By the definition of "max compaction lag", we cannot have > zero lag. So I use 0 to indicate "disable". > > > > Xiongqi (Wesley) Wu > > > On Wed, Sep 5, 2018 at 8:34 AM Colin Mc

Re: [VOTE] KIP-354 Time-based log compaction policy

> That's a fair point. We should make 0 = disable, to be consistent with the other settings. -1 is used elsewhere for disable and when seeing it in a config it's clear that it's a special meaning. 0 doesn't have to mean instant, it just means as quickly as possible. I don't think 0 is intuitive

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Might also be worth moving to a vote thread? Discussion seems to have gone as far as it can. > On 4 Sep 2018, at 12:08, xiongqi wu wrote: > > Brett, > > Yes, I will post PR tomorrow. > > Xiongqi (Wesley) Wu > > > On Sun, Sep 2, 2018 at 6:28 PM Brett Rann

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Xiongqi (Wesley) Wu > > > > > > On Thu, Aug 16, 2018 at 6:46 PM Brett Rann > > wrote: > > > >> To just clarify a bit on 1. whether there's an external storage/DB isn't > >> relevant here. > >> Compacted topics allow a tombstone record to

Re: [DISCUSS] KIP-351: Add --critical-partitions option to describe topics command

It seems to me what this is driving at is to list partitions that have no more fault-tolerance. Maybe the wording could reflect that? That rewording would have the benefit of covering 1 of 3 in sync, or 2 of 3 in sync with min.insync.replicas=2. I'm not convinced with your labelling of WARNING /

Re: [DISCUSS] KIP-354 Time-based log compaction policy

ing by > > > > setting > > > > > min dirty ratio to 0, so I decide to use "0" as disabled state. > > > > > I am ok to go with -1(disable), 0 (immediate) options. > > > > > > > > > > For the implementation, there

Re: [DISCUSS] KIP-354 Time-based log compaction policy

red in 0.10.* I wonder if there's a better way using that to get at a timestamp? On Thu, Aug 16, 2018 at 11:30 AM Brett Rann wrote: > An API was suggested by Gwen and James when I discussed it with them. For > me I can think of it as a use case for scheduling compaction rather than > relying

Re: [DISCUSS] KIP-354 Time-based log compaction policy

active segment to roll to follow the "max compaction lag" > > I can share my code so we can coordinate. > > I haven't think about a new API to force a compaction. what is the use case > for this one? > > > On Wed, Aug 15, 2018 at 5:33 PM, B

Re: [DISCUSS] KIP-354 Time-based log compaction policy

> > > > Thanks > > > > > > > > Eno > > > > > > > > > > > > > > > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu > > wrote: > > > > > > > > > Hi Kafka, > > > > > > > > > > This KIP tries to address GDPR concern to fulfill deletion request > on > > > > time > > > > > through time-based log compaction on a compaction enabled topic: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-> > > > > > 354%3A+Time-based+log+compaction+policy > > > > > > > > > > Any feedback will be appreciated. > > > > > > > > > > > > > > > Xiongqi (Wesley) Wu > > > > > > > > > > > > > > > -- Brett Rann Senior DevOps Engineer Zendesk International Ltd 395 Collins Street, Melbourne VIC 3000 Australia Mobile: +61 (0) 418 826 017

Re: [DISCUSS] KIP-354 Time-based log compaction policy

Aug 13, 2018 at 2:56 PM, Eno Thereska < > eno.there...@gmail.com> > >>> > wrote: > >>> > > >>> > > Hello, > >>> > > > >>> > > Thanks for the KIP. I'd like to see a more precise definition of > what > >>> > part > >>> > > of GDPR you are targeting as well as some sort of verification that > >>> this > >>> > > KIP actually addresses the problem. Right now I find this a bit > >>> vague: > >>> > > > >>> > > "Ability to delete a log message through compaction in a timely > >>> manner > >>> > has > >>> > > become an important requirement in some use cases (e.g., GDPR)" > >>> > > > >>> > > > >>> > > Is there any guarantee that after this KIP the GDPR problem is > >>> solved or > >>> > do > >>> > > we need to do something else as well, e.g., more KIPs? > >>> > > > >>> > > > >>> > > Thanks > >>> > > > >>> > > Eno > >>> > > > >>> > > > >>> > > > >>> > > On Thu, Aug 9, 2018 at 4:18 PM, xiongqi wu > >>> wrote: > >>> > > > >>> > > > Hi Kafka, > >>> > > > > >>> > > > This KIP tries to address GDPR concern to fulfill deletion > request > >>> on > >>> > > time > >>> > > > through time-based log compaction on a compaction enabled topic: > >>> > > > > >>> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-> > >>> > > > 354%3A+Time-based+log+compaction+policy > >>> > > > > >>> > > > Any feedback will be appreciated. > >>> > > > > >>> > > > > >>> > > > Xiongqi (Wesley) Wu > >>> > > > > >>> > > > >>> > > >>> > >> > >> > >> > >> -- > >> Xiongqi (Wesley) Wu > >> > > > > > > > > -- > > Xiongqi (Wesley) Wu > > > > > > -- > Xiongqi (Wesley) Wu > -- Brett Rann Senior DevOps Engineer Zendesk International Ltd 395 Collins Street, Melbourne VIC 3000 Australia Mobile: +61 (0) 418 826 017

Re: KAFKA-7137 - max time guarantees of compaction for GDPR et al

ut it. > One of the complications you may run into is that the active segment is not > cleaned at present. > > I went ahead and gave you wiki permission. I assumed your username is > brettrann, so let me know if it's different. > > Thanks, > Jason > > On Wed, Aug 1, 2018 a

Re: KAFKA-7137 - max time guarantees of compaction for GDPR et al

Ping on this. We're prepared to do the work i'm just not sure of the right way to start. If nobody wants to discuss it now should we just write up a KIP? On Thu, Jul 19, 2018 at 10:57 AM Brett Rann wrote: > re: https://issues.apache.org/jira/browse/KAFKA-7137 > > My team is investiga

KAFKA-7137 - max time guarantees of compaction for GDPR et al

can I get KIP access for: brettr...@gmail.com ) Thanks! -- Brett Rann Senior DevOps Engineer Zendesk International Ltd 395 Collins Street, Melbourne VIC 3000 Australia Mobile: +61 (0) 418 826 017

Re: [VOTE] 1.1.1 RC3

e.org/job/kafka-1.1-jdk7/162 > <https://builds.apache.org/job/kafka-1.1-jdk7/162> > <https://builds.apache.org/job/kafka-1.1-jdk7/162 > <https://builds.apache.org/job/kafka-1.1-jdk7/162>>* > System tests: > https://jenkins.confluent.io/job/system-test-kafka/job/1.

Re: [VOTE] 2.0.0 RC2

; http://kafka.apache.org/20/protocol.html > <http://kafka.apache.org/20/protocol.html> > > > * Successful Jenkins builds for the 2.0 branch: > > Unit/integration tests: https://builds.apache.org/job/kafka-2.0-jdk8/72/ > <https://builds.apache.org/job/kafka-2.0-jdk8/72/> >

[jira] [Created] (KAFKA-7137) ability to trigger compaction for tombstoning and GDPR

Brett Rann created KAFKA-7137: - Summary: ability to trigger compaction for tombstoning and GDPR Key: KAFKA-7137 URL: https://issues.apache.org/jira/browse/KAFKA-7137 Project: Kafka Issue Type

Re: [VOTE] 2.0.0 RC1

> > > > http://kafka.apache.org/20/documentation.html > <http://kafka.apache.org/20/documentation.html> > > > > > > > > * Protocol: > > > > http://kafka.apache.org/20/protocol.html > <http://kafka.apache.org/20/protocol.html> > >

Re: [VOTE] 1.1.0 RC4

Release artifacts to be voted upon (source and binary): > > > > > > > > > > http://home.apache.org/~rsivaram/kafka-1.1.0-rc4/ > <http://home.apache.org/~rsivaram/kafka-1.1.0-rc4/> > > > > > > > > > > > > > > > *

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations Rajini! On Thu, Jan 18, 2018 at 9:23 AM, Konstantine Karantasis < konstant...@confluent.io> wrote: > Congrats Rajini! > > -Konstantine > > On Wed, Jan 17, 2018 at 2:18 PM, Becket Qin wrote: > > > Congratulations, Rajini! > > > > On Wed, Jan 17, 2018 at 1:52

[jira] [Created] (KAFKA-6185) java.lang.OutOfMemoryError memory leak on 1.0.0 with 0.11.0.1 on disk and converting to 0.9 clients

Brett Rann created KAFKA-6185: - Summary: java.lang.OutOfMemoryError memory leak on 1.0.0 with 0.11.0.1 on disk and converting to 0.9 clients Key: KAFKA-6185 URL: https://issues.apache.org/jira/browse/KAFKA-6185