RE: [DISCUSS] KIP-382: MirrorMaker 2.0

onvention that "us-west.topic1" is a replicated topic with records originating from a remote cluster. I'm not sure I understand your concern w.r.t compacted topics and state. Can you elaborate? Ryanne On Wed, Dec 12, 2018 at 8:41 AM Michael Pearce wrote: > Ryanne, > > You haven’t desc

RE: [DISCUSS] KIP-382: MirrorMaker 2.0

ible I'd much prefer header + hops based replication rather than > lots of renamed topics. But either way, this KIP would be tremendously > useful to us so I support it all the way! :) > > On Tue, Dec 11, 2018 at 5:32 AM Michael Pearce > wrote: > > > So this is indeed wha

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

ler. Maybe file a Jira ticket to track this? Really appreciate your feedback! Ryanne On Thu, Dec 6, 2018 at 7:03 PM Michael Pearce wrote: > Re hops to stop the cycle and to allow a range of multi cluster > topologies, see https://www.rabbitmq.com/federated-exchanges.

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Re hops to stop the cycle and to allow a range of multi cluster topologies, see https://www.rabbitmq.com/federated-exchanges.html where very similar was done in rabbit. On 12/7/18, 12:47 AM, "Michael Pearce" wrote: Nice proposal. Some comments. On the section ar

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Nice proposal. Some comments. On the section around cycle detection. I would like to see support for this to be done by hops, as well e.g. using approach is to use a header for the number of hops, as the mm2 replicates it increases the hop count and you can make the mm2 configurable to only

RE: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

+1 (non-binding) I personally think the current proposed Converters described in the KIP are good, as Randall states it keeps it more in line with the pattern of the Converter methods, I think this is a good reason. -Original Message- From: Jason Gustafson [mailto:ja...@confluent.io]

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

; > >>>> > So that would solve the serialization problem. How about > connectors > > >>>> and > > >>>> > transforms that are implemented to expect a certain type of header > > >>

Re: [DISCUSS] URIs on Producer and Consumer

To me, this is a lot more in line with many other systems connections, to have the ability to have a single connection string / uri, is this really that left field suggesting or wanting this? If anything this bring kafka more standardised approach imo, to have a unified resource identifier,

Re: [DISCUSS] 2017 October release planning and release version

+1 Why not just drop the leading 0, and call next version 12.0.0 instead of 1.0.0, I think in my head I’ve always just dropped the leading 0 anyhow. On 21/07/2017, 04:15, "Neha Narkhede" wrote: +1 on 1.0. It's about time :) On Thu, Jul 20, 2017 at 5:11 PM Ewen

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

fields - but now they’d have a choice). > > If people like this idea then I can create a Jira and a PR. (Would a KIP be required also?). If people don’t, I’ll move along quietly… > > Thanks, > > Andy > > On 19/05/2017, 08:27, "Michael Pearce" <michael.pea...@ig.com&

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

Just copying in from the mail list, another use case / +1 for the builder pattern, so its kept with the KIP thread. On 18/05/2017, 22:06, "Andrew Coates" wrote: Thanks Mike On Thu, 18 May 2017 at 21:33, Michael André Pearce <

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

+1 From: Joel Koshy Sent: Friday, May 5, 2017 4:32:42 AM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches. +1 On Thu, May 4, 2017 at 7:00 PM Becket Qin

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Hi Ewen, Did you get a chance to look at the updated sample showing the idea? Did it help? Cheers Mike Sent using OWA for iPhone From: Michael Pearce <michael.pea...@ig.com> Sent: Wednesday, May 3, 2017 10:11:55 AM To: dev@kafka.apache.org Subje

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

; > flexible, most people default to using StringConverter for the serializer > > and Connectors will end up defaulting to that just for compatibility... > > > > I'm not sure I have a real proposal yet, but I do think understanding the > > impact of using

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

On Sat, Apr 29, 2017 at 12:12 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi All, > > Now KIP-82 is committed I would like to discuss extending the work to > expose it in Kafka Connect, its primary focus being so connectors that may > do similar tasks as MirrorMakers, eithe

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

s just driven by the docs. I’m okay with that, but we need concensus On 1/5/17, 6:08 am, "Michael Pearce" <michael.pea...@ig.com> wrote: Why not, instead of deprecating or removing whats there, as noted, its a point of preference, think about something that could wrap the existin

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

Why not, instead of deprecating or removing whats there, as noted, its a point of preference, think about something that could wrap the existing, but provide an api that for you is cleaner? e.g. here's a sample idea building on a fluent api way. (this wraps the producer and producer records so

Re: KIP-82 Record Headers Update

for my next KIP ;) – please see KIP-145. Thanks to all, Mike From: Michael Pearce <michael.pea...@ig.com> Date: Friday, 31 March 2017 at 23:19 To: "dev@kafka.apache.org" <dev@kafka.apache.org> Subject: KIP-82 Record Headers Update Hi All, PR to add the client api building o

[DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Hi All, Now KIP-82 is committed I would like to discuss extending the work to expose it in Kafka Connect, its primary focus being so connectors that may do similar tasks as MirrorMakers, either Kafka->Kafka or JMS-Kafka would be able to replicate the headers. It would be ideal but not

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

This is ref to this section that's still there " The ProducerRecord constructors will all be deprecated, except the main long explicit one: " Sent using OWA for iPhone ____ From: Michael Pearce <michael.pea...@ig.com> Sent: Saturday, April 29,

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

Probably should remove the words in the kip the other constructors are going to be deprecated if they aren't any more with just the tactical change Sent using OWA for iPhone From: Michael Pearce <michael.pea...@ig.com> Sent: Saturday, April 29, 201

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

P here: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69406838 Added constructors here: https://github.com/apache/kafka/pull/2894 Let me know your thoughts, Regards, Stephane On 25/4/17, 2:58 am, "Michael Pearce" <michael.pea...@ig.com> wrote: Why not simply

Re: [VOTE] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs

+1 From: Colin McCabe Sent: Saturday, April 29, 2017 1:09:25 AM To: dev@kafka.apache.org Subject: [VOTE] KIP-140: Add administrative RPCs for adding, deleting, and listing ACLs Hi all, I'd like to start the voting for KIP-140: Add

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

Why not simply make a cleaner client fluent API wrapper? The internals use and send via current api, but provide a cleaner more fluent api. A good example here is HTTP compontents where they did this. https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html On 24/04/2017,

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

If moving to a wither pattern instead of a builder. How will this enforce immutability? Eg current PR it is now changing to allow possible change values once set. Or are you proposing to change it to a mutable record? And move to a closable record similar to the closing of the headers on send.

KIP-82 Record Headers Update

Hi All, PR to add the client api building on the protocol changes already in trunk/master is raised in GitHub https://github.com/apache/kafka/pull/2772 Thanks Jeroen for feedback already :). Please note we have dropped Java 8 bits for now and replaced with Java 7 compatible until KIP-118 is

Re: [VOTE] KIP-82 Add Record Headers

ion. Thanks! Ismael On Fri, Mar 24, 2017 at 8:29 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jun, > > Thanks for your vote, I’ve updated the wiki to document this detail. > > Cheers > Mike > > On 24/03/2017, 05:00, "Jun Rao" <j...@confluent.io&

Re: [VOTE] KIP-82 Add Record Headers

Renu Tewari Jeroen van Disseldorp Michael Pearce Non Binding -1: 0 Non binding +0: 0 Based on the above result, we now have a lazy majority I am now closing this voting thread as accepted. I will move the KIP to accepted, and start work on implementation. As per KIP discussion Jason has

Re: [VOTE] KIP-82 Add Record Headers

hod". However, there is no close() > in > > Headers. > > > > Thanks, > > > > Jun > > > > On Thu, Mar 23, 2017 at 9:34 AM, Michael Pearce <michael.pea...@ig.com> > > wrote: > > > > > Thanks all fo

Re: [VOTE] KIP-82 Add Record Headers

-coding > >> things. I > >> think the type should instead by [Key Value] in our BNF, where key > and > >> value are both short strings as used elsewhere. This brings it in > >> line with > >> the rest of the protocol.

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Ismael, Sorry, The response below was in regards to your comments, got my wires crossed, apologies. Hi Jun, I’m happy with the change, I see Jason updated our KIP, many thanks for this, and thanks for implementing for us ☺ Cheers Mike On 20/03/2017, 13:19, "Michael P

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Jun, Thanks the comments I’ve updated the KIP a little where agreement. My comments: 1) Good point, removed from the interface. See updated KIP 2) I think, Radai’s suggested header(String key) is a cleaner method name, but happy to change if community believe lastHeader is better. I’ll keep

Re: [DISCUSS] KIP-82 - Add Record Headers

n succeeded? Cheers Mike ________ From: Michael Pearce Sent: Wednesday, March 1, 2017 5:55 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-82 - Add Record Headers Hi Radai: RE: Header header(String key) - returns JUST ONE (the very last) value given a

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Radai: RE: Header header(String key) - returns JUST ONE (the very last) value given a key Iterable headers(String key) - returns ALL under a key void add(Header header) - appends a header (key inside). void remove(String key) - removes ALL HEADERS under a key. I don't think this one is

Re: [DISCUSS] KIP-82 - Add Record Headers

d(new Header("someKey", value)); > > this is why i think we should start with get()/set() which are single-value > map semantics (so set overwrites), then add getAll() (multi-map), append() > etc on top. make the common case pretty. > > On Sun, Feb 26, 2017 at 2:01 A

Re: [DISCUSS] KIP-82 - Add Record Headers

this KIP is already complicated, I would rather leave this out of the scope and address that later when needed, e.g. after having batch level interceptors. Thanks, Jiangjie (Becket) Qin On Fri, Feb 24, 2017 at 3:56 PM, Michael Pearce <michael.pea...@ig.com> wrote: > KIP updated in r

Re: [DISCUSS] KIP-82 - Add Record Headers

oduceRequest. Can you change this to say that the changes will > piggyback > onto V3 of ProduceRequest and V4 of FetchRequest which were introduced > in > KIP-98? On 24/02/2017, 23:20, "Michael Pearce" <michael.pea...@ig.com> wrote: We’re

Re: [DISCUSS] KIP-82 - Add Record Headers

ses? Thanks, Jason On Fri, Feb 24, 2017 at 2:22 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jason, > > Sorry I thought this was the agreed compromise to provide an api that > avoid boiler plate in return for immutabilty. > > If not

Re: [DISCUSS] KIP-82 - Add Record Headers

quest which were introduced in KIP-98? The rest of the KIP looks good to me. -Jason On Fri, Feb 24, 2017 at 12:46 PM, Michael Pearce <michael.pea...@ig.com> wrote: > I’ve added the methods on the ProducerRecord that will return a new > instance of Producer

Re: [DISCUSS] KIP-82 - Add Record Headers

I’ve added the methods on the ProducerRecord that will return a new instance of ProducerRecord with modified headers. On 24/02/2017, 19:22, "Michael Pearce" <michael.pea...@ig.com> wrote: Pattern.compile is expensive, and even if cached String.equals is faster than matched

Re: [DISCUSS] KIP-82 - Add Record Headers

Ismael noted, for common scenarios it is possible to get reasonable performance with immutable objects. -Jason On Fri, Feb 24, 2017 at 8:48 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi > > On 1, How can you guarantee two separate implemented clients would add > the h

Re: [DISCUSS] KIP-82 - Add Record Headers

ds little memory overhead if we have Header instances is to simply add a self-reference to the previous Header in the linked list. Ismael On Thu, Feb 23, 2017 at 1:09 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Im happy to compromise to keep it mutable but mo

Re: [DISCUSS] KIP-82 - Add Record Headers

or strip it out i'd need to copy the whole record. On Wed, Feb 22, 2017 at 5:09 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Im happy to compromise to keep it mutable but move to an append style api. > (as in guava interables concat) > > cl

Re: [DISCUSS] KIP-82 - Add Record Headers

On Wed, Feb 22, 2017 at 3:03 PM, Michael Pearce <michael.pea...@ig.com> wrote: > I wasn’t referring to the headers needing to be copied, im meaning the > fact we’d be forcing a new producer record to be created, with all the > contents copied. > > i

Re: [DISCUSS] KIP-82 - Add Record Headers

call. On 22/02/2017, 22:57, "Michael Pearce" <michael.pea...@ig.com> wrote: Lazy init can achieve/avoid that. Re the concat, why don’t we implement that inside the Headers rather than causing everyone to implement this as adding headers in interceptors will be a domin

Re: [DISCUSS] KIP-82 - Add Record Headers

sn't necessarily mean that they need to be copied: you can do a trick like Guava's Iterables.concat to add additional headers without changing the underlying collections. -Jason On Wed, Feb 22, 2017 at 2:22 PM, Michael Pearce <michael.pea...@ig.com> wrote: > If th

Re: [DISCUSS] KIP-82 - Add Record Headers

rRecord with a new header added to the current headers and returning it. Would that not work? -Jason On Wed, Feb 22, 2017 at 1:45 PM, Michael Pearce <michael.pea...@ig.com> wrote: > So how would you have this work if not mutable where interceptors would > add headers?

Re: [DISCUSS] KIP-82 - Add Record Headers

nging the interface). > > 3. Do we need the `Headers.get()` method? People usually assume that > `get` > would be efficient, but depending on the implementation (the current > proposal states that an array would be used), it may not be. If we > expect > the number

Re: [DISCUSS] KIP-82 - Add Record Headers

ders to be small, it doesn't matter though. Ismael On Tue, Feb 21, 2017 at 6:38 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jason, > > Have converted the interface/api bullets into interface code snippets. > > Agreed implementation won’t

Re: [DISCUSS] KIP-82 - Add Record Headers

Hi Jason, Have converted the interface/api bullets into interface code snippets. Agreed implementation won’t take too long. We have early versions already. Maybe a week before you think about merging I would assume it would be more stabilised? I was thinking then we could fork from your

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

" <ja...@confluent.io> wrote: Done. I've left the key and value as optional since we may not have reached consensus on whether to use attributes or not. Perhaps we should just keep it simple and not do it? The benefit seems small. -Jason On Tue, Feb 21, 2017 at 4:05 PM

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

ther add it back and put headers at the end. This is more consistent with the rest of the Kafka protocol. -Jason On Tue, Feb 21, 2017 at 3:58 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Or we keep as is (valuelen removed), and headers are added with headers >

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

deduce the value length. However, if we are adding record headers to the end, we would need to introduce the value length along with that change. On Tue, Feb 21, 2017 at 3:34 PM, Michael Pearce <michael.pea...@ig.com> wrote: > It seems I cannot add comment on the doc.

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

gt; > > I have only one minor question, in the wiki (and the doc) the new > message > > > set format has a "FirstSequence" field, should it just be "Sequence" if > > the > > > sequence is always associated with a message set? > >

Re: [DISCUSS] KIP-82 - Add Record Headers

ject On 19/02/2017, 20:06, "Michael Pearce" <michael.pea...@ig.com> wrote: On points 1 and 2 I agree. This also affects kip-98, I should expect this resolved before that vote also passes. If it is accepted there (I’m assuming this is getting discussed on that KIP? As you

Re: [DISCUSS] KIP-82 - Add Record Headers

shuffling data into/out-of kafka > (kafka connect or equivalent) where metadata from one end could copied over > to the other (S3, for example uses http headers for user-accessible > metadata). it will be kafka client code getting/setting those headers. not > an

Re: [DISCUSS] KIP-82 - Add Record Headers

is used to continue > tracing. I still don't understand the point about batching. The consumer records are exposed as a batch in the consumer interceptor, but you can still iterate them individually. It is no different for the consumer API itself. -Jason On Fri, F

Re: [DISCUSS] KIP-82 - Add Record Headers

[] with length value makes this a lot easier to skip. On 17/02/2017, 20:37, "Michael Pearce" <michael.pea...@ig.com> wrote: What’s the issue with exposing a method getHeaders on the producer/consumer record? It doesn’t break anything. We don’t need any special version.

Re: [DISCUSS] KIP-82 - Add Record Headers

allenge is for MM-like use cases. Let me see if I can come up with a concrete proposal for that problem. -Jason On Fri, Feb 17, 2017 at 11:55 AM, Michael Pearce <michael.pea...@ig.com> wrote: > I am happy to move the definition of the header into the message

Re: [DISCUSS] KIP-82 - Add Record Headers

lazily to avoid parsing out all the headers into little objects. HashMaps themselves are kind of expensive and the consumer is very perf sensitive so and making gazillions of hashmaps that may or may not get used is probably a bad idea.” On 17/02/2017, 19:44, "Michael P

Re: [DISCUSS] KIP-82 - Add Record Headers

challenge in my mind is figuring out how an MM use case would work. It > > would be more cumbersome to replicate headers through an interceptor, > > though arguably MM should be working at a lower level anyway. > > > > -Jason > > > > On Fri, Feb 17, 2017 at 10:16 AM

Re: [DISCUSS] KIP-82 - Add Record Headers

e would rather have them implement headers directly in their own schema. Supposing for the sake of argument that it was possible, my question is whether it be sufficient to expose the headers in the interceptor API and not in the common API? -Jason On Fri, Feb 17, 2017 at 3:26 AM, Michael Pearce <m

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

+0 I think need some unified agreement on the VarInts. Would this also change in all other area’s of the protocol, e.g. value and key length in message protocol, to keep this uniform across all protocols going forwards? On 17/02/2017, 00:23, "Apurva Mehta" wrote:

Re: [DISCUSS] KIP-82 - Add Record Headers

, "Michael Pearce" <michael.pea...@ig.com> wrote: On the point re: headers in the message protocol being a byte array and not a count of elements followed by the elements. Again this was discussed/argued previously. It was agreed on for a few reasons some of which yo

Re: [DISCUSS] KIP-82 - Add Record Headers

; then the need to use attributes becomes quite a bit weaker, right? The >> >> other thing I find slightly odd is the fact that null headers has no >> >> actual >> >> semantic meaning for the message (unlike null keys and values). It is >>

Re: [DISCUSS] KIP-82 - Add Record Headers

2 iterations of the "type > definition DSL") > > 2. this is an implementation detail, and not even a very "user facing" one? > to the best of my understanding the vote process is on proposed > API/behaviour. also - since we're willing to go wit

Re: [VOTE] KIP-82 Add Record Headers

o - a more usable link to the discussion thread: > > http://markmail.org/message/x5wlkieexinovsz3 > > > > On Tue, Feb 14, 2017 at 9:42 AM, Michael Pearce <michael.pea...@ig.com> > > wrote: > > > > > Hi all, > > > >

[VOTE] KIP-82 Add Record Headers

Hi all, We would like to start the voting process for KIP-82 – Add record headers. The KIP can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers Discussion thread(s) can be found here:

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Kreps <j...@confluent.io> wrote: > >> Makes sense! >> >> -Jay >> >> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <michael.pea...@ig.com> >> wrote: >> >> > Wow just read that def over tired. Hopefully it makes sense. Or you get >>

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Wow just read that def over tired. Hopefully it makes sense. Or you get the gist at least. From: Michael Pearce <michael.pea...@ig.com> Sent: Monday, December 19, 2016 9:19:02 PM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-87 - Add Comp

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

:16 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jay > > I disagree here that we are breaking any compatibility, we went through > this on the discussion thread in fact with the help of that thread is how > the kip came to the solution. > > Also on the supported c

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

know we have three others lined up. I hope that explains my reasoning. Does it make sense at all? Ismael On Fri, Dec 16, 2016 at 1:53 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Ismael > > > So I understand what is being suggested, is we allow on the ProducerRe

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

t ideal, but would limit the complexity of maintaining compatibility while providing a path for the (what I think) are the right semantics. Ismael On Tue, Dec 13, 2016 at 8:32 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Ismael > > Did you see

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

our code in some environment seems worse to me then where we currently are. I think the question is how would this combination be explained to users and does it make sense? -Jay On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Chaps, > > Can we either g

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Chaps, Can we either get one more +1 binding (we have 2 already) on the existing? Or have some response on the below possible alternative. We are keen to get working on this, so we make next feature release. Cheers Mike On 13/12/2016, 16:32, "Michael Pearce" <michael.pea...@

Re: [VOTE] 0.10.1.1 RC0

Is there any update on this, when do we expect and RC1 for vote? We see KAFKA-4497 is marked Resolved. Cheers Mike On 13/12/2016, 00:59, "Guozhang Wang" wrote: I see. Currently the upload command has not included the 2.12 version yet, I will manually do that in the

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

licy), but downstream applications > might care to know the difference between a delete and a null value. In > fact both versions of the same log (compacted and time-retention) could be > useful and I don't think it'll be uncommon to maintain both or use KIP-71 > to maintain a hybrid co

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

and mechanics? @Mayuresh, Radai, Becket, Jun as you’ve all +1 voted on the existing solution to do this by upgrading the existing compaction policy would any of you have concerns with the alternative? Regards, Mike On 11/12/2016, 23:33, "Michael Pearce" <michael.pea...@ig.com> wro

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

n both or use KIP-71 to maintain a hybrid compacted/retention topic. -Ewen On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jay, > > Why wouldn't that work, the tombstone value is only looked at by the > broker, on a topic configured for compaction

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

topics that use null as an acceptable value sent by the producer and expected by the consumer. -Jay On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Ewen, > > I think the easiest way to show this is with code. > > As you can see we keep the

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

ptional schemas in other formats), e.g. [null, string], in which case losing the schema truly is losing information (whereas null is already the only valid value for a pure null schema). -Ewen On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <michael.pea...@ig.com> wrote: >

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

at use this feature and send null to mean delete. It seems like this would break compatibility with them, and they would have to be rewritten to right? -Jay On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jun, > > 4) On v3 we honour the tombsto

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Apologies on the spelling. *Hi Jay, From: Michael Pearce <michael.pea...@ig.com> Sent: Friday, December 9, 2016 7:52:25 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging Hi Jey 1) I

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

plementation for pluggable multi-system XA. -Jay On Fri, Dec 9, 2016 at 10:25 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jay, > > I can't go too deep into exact implantation due to no NDA. So apologies > here. > > Essentially we have multiple processes each

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

2016 at 10:19 PM, Michael Pearce <michael.pea...@ig.com> wrote: > Usecase in IG: > > Fund transfer between accounts. When we debit one account and fund another > we must ensure the records to both occur as an acid action, and as a single > transaction. > > Today we achieve

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

ations to be resilient to message duplication and loss, rather > than tightly coupling resource managers and ending up with something fragile. > > Don't get me wrong. This is not an anti-Kafka rant. I just work with people > used to traditional transactional systems, making use of Kafka for

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

the accounts. To move this flow to Kafka we would need support of XA transaction. Sent using OWA for iPhone From: Michael Pearce <michael.pea...@ig.com> Sent: Friday, December 9, 2016 6:09:06 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS]

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Hi Jay, For me having an XA transaction allows for ensuring ACID across my application. I believe it is part of the JMS api, and obviously JMS still is in enterprise very widely adopted for Messaging transport , so obviously to say it isn't widely used i think is ignoring a whole range of

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

Like wise I think handling transactions separately is important as we should consider supporting or at least a how in future session transactions, xa transactions etc Sent using OWA for iPhone From: Michael Pearce <michael.pea...@ig.com> Sent: Th

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

he two essential ingredients needed to complete the exactly-once story, so we prefer to keep them together. -Jason On Thu, Dec 8, 2016 at 12:36 AM, Michael Pearce <michael.pea...@ig.com> wrote: > My apologies, my computer is auto spellchecking and I didn’t notice before > sending. >

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

My apologies, my computer is auto spellchecking and I didn’t notice before sending. *@Sriram Thanks Mike On 08/12/2016, 08:35, "Michael Pearce" <michael.pea...@ig.com> wrote: @Shiram, I would like to be able to have though exactly once delivery without the need t

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

since there were interdependencies between the concepts. On Tue, Dec 6, 2016 at 9:04 AM, Michael Pearce <michael.pea...@ig.com> wrote: > For dealing with exactly once delivery. > > As an alternative option has it been considered to have a message uuid in >

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

? 4.4 Do 4.1, 4.2, 4.3 depend on whether the topic is compacted on not? Thanks, Jun On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Not at all. This only acts on compacted topics just as what occurs today >

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

bstone Flag Hi, Michael, 4. Hmm, does that mean the new client library can never send a null message even to a regular topic? This seems like a change of the existing behavior. Thanks, Jun On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jun, > >

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

6, 2016 at 8:54 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jun > > do we have your vote on this now? > > Any other concerns? > > Cheers > Mike > > Sent using OWA for iPhone > ____ > From: Michael Pearce <m

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

For dealing with exactly once delivery. As an alternative option has it been considered to have a message uuid in the record, that then is deduped on consumption? Similar to https://activemq.apache.org/artemis/docs/1.0.0/duplicate-detection.html Agreed this does not deal with transaction

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

Hi Jun do we have your vote on this now? Any other concerns? Cheers Mike Sent using OWA for iPhone From: Michael Pearce <michael.pea...@ig.com> Sent: Saturday, December 3, 2016 1:37:45 AM To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-87

Re: [DISCUSS] KIP-82 - Add Record Headers

From: James Cheng <wushuja...@gmail.com> Sent: Monday, December 5, 2016 8:50:30 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-82 - Add Record Headers > On Dec 2, 2016, at 4:57 PM, Michael Pearce <michael.pea...@ig.com> wrote: > > Hi Jun. > > RE Mirr

Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag

2, 2016 at 3:19 AM, Michael Pearce <michael.pea...@ig.com> wrote: > Hi Jao > > Thanks for the response. Sorry for slow reply, both with personal sickness > and also battling some critical issues encountered since upgrading to > 0.10.1.0 >

Re: [DISCUSS] KIP-82 - Add Record Headers

; >> > >> >> >>> > idea in general, so we can move ahead and try to agree > on > > the > > >> > >> >> details? > > >> > >> >> >>> > > > >> > >> >> >>>

  1   2   >