For what it is worth also i agree. As a user:
1) Yes - Headers are worthwhile
2) Yes - Headers should be a top level option
14.11.2016, 21:15, "Ignacio Solis" <iso...@igso.net>:
> 1) Yes - Headers are worthwhile
> 2) Yes - Headers should be a top level option
>
> On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce <michael.pea...@ig.com>
> wrote:
>
>> Hi Roger,
>>
>> The kip details/examples the original proposal for key spacing , not the
>> new mentioned as per discussion namespace idea.
>>
>> We will need to update the kip, when we get agreement this is a better
>> approach (which seems to be the case if I have understood the general
>> feeling in the conversation)
>>
>> Re the variable ints, at very early stage we did think about this. I think
>> the added complexity for the saving isn't worth it. I'd rather go with, if
>> we want to reduce overheads and size int16 (2bytes) keys as it keeps it
>> simple.
>>
>> On the note of no headers, there is as per the kip as we use an attribute
>> bit to denote if headers are present or not as such provides a zero
>> overhead currently if headers are not used.
>>
>> I think as radai mentions would be good first if we can get clarity if do
>> we now have general consensus that (1) headers are worthwhile and useful,
>> and (2) we want it as a top level entity.
>>
>> Just to state the obvious i believe (1) headers are worthwhile and (2)
>> agree as a top level entity.
>>
>> Cheers
>> Mike
>> ________________________________________
>> From: Roger Hoover <roger.hoo...@gmail.com>
>> Sent: Wednesday, November 9, 2016 9:10:47 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>>
>> Sorry for going a little in the weeds but thanks for the replies regarding
>> varint.
>>
>> Agreed that a prefix and {int, int} can be the same. It doesn't look like
>> that's what the KIP is saying the "Open" section. The example shows
>> 2100001
>> for New Relic and 210002 for App Dynamics implying that the New Relic
>> organization will have only a single header id to work with. Or is 2100001
>> a prefix? The main point of a namespace or prefix is to reduce the
>> overhead of config mapping or registration depending on how
>> namespaces/prefixes are managed.
>>
>> Would love to hear more feedback on the higher-level questions though...
>>
>> Cheers,
>>
>> Roger
>>
>> On Wed, Nov 9, 2016 at 11:38 AM, radai <radai.rosenbl...@gmail.com> wrote:
>>
>> > I think this discussion is getting a bit into the weeds on technical
>> > implementation details.
>> > I'd liek to step back a minute and try and establish where we are in the
>> > larger picture:
>> >
>> > (re-wording nacho's last paragraph)
>> > 1. are we all in agreement that headers are a worthwhile and useful
>> > addition to have? this was contested early on
>> > 2. are we all in agreement on headers as top level entity vs headers
>> > squirreled-away in V?
>> >
>> > if there are still concerns around these #2 points (#jay? #jun?)?
>> >
>> > (and now back to our normal programming ...)
>> >
>> > varints are nice. having said that, its adding complexity (see
>> > https://github.com/addthis/stream-lib/blob/master/src/
>> > main/java/com/clearspring/analytics/util/Varint.java
>> > as 1st google result) and would require anyone writing other clients (C?
>> > Python? Go? Bash? ;-) ) to get/implement the same, and for relatively
>> > little gain (int vs string is order of magnitude, this isnt).
>> >
>> > int namespacing vs {int, int} namespacing are basically the same thing -
>> > youre just namespacing an int64 and giving people while 2^32 ranges at a
>> > time. the part i like about this is letting people have a large swath of
>> > numbers with one registration so they dont have to come back for every
>> > single plugin/header they want to "reserve".
>> >
>> >
>> > On Wed, Nov 9, 2016 at 11:01 AM, Roger Hoover <roger.hoo...@gmail.com>
>> > wrote:
>> >
>> > > Since some of the debate has been about overhead + performance, I'm
>> > > wondering if we have considered a varint encoding (
>> > > https://developers.google.com/protocol-buffers/docs/encoding#varints)
>> > for
>> > > the header length field (int32 in the proposal) and for header ids? If
>> > you
>> > > don't use headers, the overhead would be a single byte and for each
>> > header
>> > > id < 128 would also need only a single byte?
>> > >
>> > >
>> > >
>> > > On Wed, Nov 9, 2016 at 6:43 AM, radai <radai.rosenbl...@gmail.com>
>> > wrote:
>> > >
>> > > > @magnus - and very dangerous (youre essentially downloading and
>> > executing
>> > > > arbitrary code off the internet on your servers ... bad idea without
>> a
>> > > > sandbox, even with)
>> > > >
>> > > > as for it being a purely administrative task - i disagree.
>> > > >
>> > > > i wish it would, really, because then my earlier point on the
>> > complexity
>> > > of
>> > > > the remapping process would be invalid, but at linkedin, for example,
>> > we
>> > > > (the team im in) run kafka as a service. we dont really know what our
>> > > users
>> > > > (developing applications that use kafka) are up to at any given
>> moment.
>> > > it
>> > > > is very possible (given the existance of headers and a corresponding
>> > > plugin
>> > > > ecosystem) for some application to "equip" their producers and
>> > consumers
>> > > > with the required plugin without us knowing. i dont mean to imply
>> thats
>> > > > bad, i just want to make the point that its not as simple keeping it
>> in
>> > > > sync across a large-enough organization.
>> > > >
>> > > >
>> > > > On Wed, Nov 9, 2016 at 6:17 AM, Magnus Edenhill <mag...@edenhill.se>
>> > > > wrote:
>> > > >
>> > > > > I think there is a piece missing in the Strings discussion, where
>> > > > > pro-Stringers
>> > > > > reason that by providing unique string identifiers for each header
>> > > > > everything will just
>> > > > > magically work for all parts of the stream pipeline.
>> > > > >
>> > > > > But the strings dont mean anything by themselves, and while we
>> could
>> > > > > probably envision
>> > > > > some auto plugin loader that downloads, compiles, links and runs
>> > > plugins
>> > > > > on-demand
>> > > > > as soon as they're seen by a consumer, I dont really see a use-case
>> > for
>> > > > > something
>> > > > > so dynamic (and fragile) in practice.
>> > > > >
>> > > > > In the real world an application will be configured with a set of
>> > > plugins
>> > > > > to either add (producer)
>> > > > > or read (consumer) headers.
>> > > > > This is an administrative task based on what features a client
>> > > > > needs/provides and results in
>> > > > > some sort of configuration to enable and configure the desired
>> > plugins.
>> > > > >
>> > > > > Since this needs to be kept somewhat in sync across an organisation
>> > > > (there
>> > > > > is no point in having producers
>> > > > > add headers no consumers will read, and vice versa), the added
>> > > complexity
>> > > > > of assigning an id namespace
>> > > > > for each plugin as it is being configured should be tolerable.
>> > > > >
>> > > > >
>> > > > > /Magnus
>> > > > >
>> > > > > 2016-11-09 13:06 GMT+01:00 Michael Pearce <michael.pea...@ig.com>:
>> > > > >
>> > > > > > Just following/catching up on what seems to be an active night :)
>> > > > > >
>> > > > > > @Radai sorry if it may seem obvious but what does MD stand for?
>> > > > > >
>> > > > > > My take on String vs Int:
>> > > > > >
>> > > > > > I will state first I am pro Int (16 or 32).
>> > > > > >
>> > > > > > I do though playing devils advocate see a big plus with the
>> > argument
>> > > of
>> > > > > > String keys, this is around integrating into an existing
>> > eco-system.
>> > > > > >
>> > > > > > As many other systems use String based headers (Flume, JMS) it
>> > makes
>> > > > it
>> > > > > > much easier for these to be incorporated/integrated into.
>> > > > > >
>> > > > > > How with Int based headers could we provide a way/guidence to
>> make
>> > > this
>> > > > > > integration simple / easy with transition flows over to kafka?
>> > > > > >
>> > > > > > * tough luck buddy you're on your own
>> > > > > > * simply hash the string into int code and hope for no collisions
>> > > (how
>> > > > to
>> > > > > > convert back though?)
>> > > > > > * http2 style as mentioned by nacho.
>> > > > > >
>> > > > > > cheers,
>> > > > > > Mike
>> > > > > >
>> > > > > >
>> > > > > > ________________________________________
>> > > > > > From: radai <radai.rosenbl...@gmail.com>
>> > > > > > Sent: Wednesday, November 9, 2016 8:12 AM
>> > > > > > To: dev@kafka.apache.org
>> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>> > > > > >
>> > > > > > thinking about it some more, the best way to transmit the header
>> > > > > remapping
>> > > > > > data to consumers would be to put it in the MD response payload,
>> so
>> > > > maybe
>> > > > > > it should be discussed now.
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Nov 9, 2016 at 12:09 AM, radai <
>> radai.rosenbl...@gmail.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > im not opposed to the idea of namespace mapping. all im saying
>> is
>> > > > that
>> > > > > > its
>> > > > > > > not part of the "mvp" and, since it requires no wire format
>> > change,
>> > > > can
>> > > > > > > always be added later.
>> > > > > > > also, its not as simple as just configuring MM to do the
>> > transform:
>> > > > > lets
>> > > > > > > say i've implemented large message support as {666,1} and on
>> some
>> > > > > mirror
>> > > > > > > target cluster its been remapped to {999,1}. the consumer
>> plugin
>> > > code
>> > > > > > would
>> > > > > > > also need to be told to look for the large message "part X of
>> Y"
>> > > > header
>> > > > > > > under {999,1}. doable, but tricky.
>> > > > > > >
>> > > > > > > On Tue, Nov 8, 2016 at 10:29 PM, Gwen Shapira <
>> g...@confluent.io
>> > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > >> While you can do whatever you want with a namespace and your
>> > code,
>> > > > > > >> what I'd expect is for each app to namespaces configurable...
>> > > > > > >>
>> > > > > > >> So if I accidentally used 666 for my HR department, and still
>> > want
>> > > > to
>> > > > > > >> run RadaiApp, I can config "namespace=42" for RadaiApp and
>> > > > everything
>> > > > > > >> will look normal.
>> > > > > > >>
>> > > > > > >> This means you only need to sync usage inside your own
>> > > organization.
>> > > > > > >> Still hard, but somewhat easier than syncing with the entire
>> > > world.
>> > > > > > >>
>> > > > > > >> On Tue, Nov 8, 2016 at 10:07 PM, radai <
>> > > radai.rosenbl...@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >> > and we can start with {namespace, id} and no re-mapping
>> > support
>> > > > and
>> > > > > > >> always
>> > > > > > >> > add it later on if/when collisions actually happen (i dont
>> > think
>> > > > > > they'd
>> > > > > > >> be
>> > > > > > >> > a problem).
>> > > > > > >> >
>> > > > > > >> > every interested party (so orgs or individuals) could then
>> > > > register
>> > > > > a
>> > > > > > >> > prefix (0 = reserved, 1 = confluent ... 666 = me :-) ) and
>> do
>> > > > > whatever
>> > > > > > >> with
>> > > > > > >> > the 2nd ID - so once linkedin registers, say 3, then
>> linkedin
>> > > devs
>> > > > > are
>> > > > > > >> free
>> > > > > > >> > to use {3, *} with a reasonable expectation to to collide
>> with
>> > > > > > anything
>> > > > > > >> > else. further partitioning of that * becomes linkedin's
>> > problem,
>> > > > but
>> > > > > > the
>> > > > > > >> > "upstream registration" of a namespace only has to happen
>> > once.
>> > > > > > >> >
>> > > > > > >> > On Tue, Nov 8, 2016 at 9:03 PM, James Cheng <
>> > > wushuja...@gmail.com
>> > > > >
>> > > > > > >> wrote:
>> > > > > > >> >
>> > > > > > >> >>
>> > > > > > >> >>
>> > > > > > >> >>
>> > > > > > >> >> > On Nov 8, 2016, at 5:54 PM, Gwen Shapira <
>> > g...@confluent.io>
>> > > > > > wrote:
>> > > > > > >> >> >
>> > > > > > >> >> > Thank you so much for this clear and fair summary of the
>> > > > > arguments.
>> > > > > > >> >> >
>> > > > > > >> >> > I'm in favor of ints. Not a deal-breaker, but in favor.
>> > > > > > >> >> >
>> > > > > > >> >> > Even more in favor of Magnus's decentralized suggestion
>> > with
>> > > > > > Roger's
>> > > > > > >> >> > tweak: add a namespace for headers. This will allow each
>> > app
>> > > to
>> > > > > > just
>> > > > > > >> >> > use whatever IDs it wants internally, and then let the
>> > admin
>> > > > > > >> deploying
>> > > > > > >> >> > the app figure out an available namespace ID for the app
>> to
>> > > > live
>> > > > > > in.
>> > > > > > >> >> > So io.confluent.schema-registry can be namespace 0x01 on
>> my
>> > > > > > >> deployment
>> > > > > > >> >> > and 0x57 on yours, and the poor guys developing the app
>> > don't
>> > > > > need
>> > > > > > to
>> > > > > > >> >> > worry about that.
>> > > > > > >> >> >
>> > > > > > >> >>
>> > > > > > >> >> Gwen, if I understand your example right, an application
>> > > deployer
>> > > > > > might
>> > > > > > >> >> decide to use 0x01 in one deployment, and that means that
>> > once
>> > > > the
>> > > > > > >> message
>> > > > > > >> >> is written into the broker, it will be saved on the broker
>> > with
>> > > > > that
>> > > > > > >> >> specific namespace (0x01).
>> > > > > > >> >>
>> > > > > > >> >> If you were to mirror that message into another cluster,
>> the
>> > > 0x01
>> > > > > > would
>> > > > > > >> >> accompany the message, right? What if the deployers of the
>> > same
>> > > > app
>> > > > > > in
>> > > > > > >> the
>> > > > > > >> >> other cluster uses 0x57? They won't understand each other?
>> > > > > > >> >>
>> > > > > > >> >> I'm not sure that's an avoidable problem. I think it simply
>> > > means
>> > > > > > that
>> > > > > > >> in
>> > > > > > >> >> order to share data, you have to also have a shared (agreed
>> > > upon)
>> > > > > > >> >> understanding of what the namespaces mean. Which I think
>> > makes
>> > > > > sense,
>> > > > > > >> >> because the alternate (sharing *nothing* at all) would mean
>> > > that
>> > > > > > there
>> > > > > > >> >> would be no way to understand each other.
>> > > > > > >> >>
>> > > > > > >> >> -James
>> > > > > > >> >>
>> > > > > > >> >> > Gwen
>> > > > > > >> >> >
>> > > > > > >> >> > On Tue, Nov 8, 2016 at 4:23 PM, radai <
>> > > > > radai.rosenbl...@gmail.com>
>> > > > > > >> >> wrote:
>> > > > > > >> >> >> +1 for sean's document. it covers pretty much all the
>> > > > trade-offs
>> > > > > > and
>> > > > > > >> >> >> provides concrete figures to argue about :-)
>> > > > > > >> >> >> (nit-picking - used the same xkcd twice, also trove has
>> > been
>> > > > > > >> superceded
>> > > > > > >> >> for
>> > > > > > >> >> >> purposes of high performance collections: look at
>> > > > > > >> >> >> https://github.com/leventov/Koloboke)
>> > > > > > >> >> >>
>> > > > > > >> >> >> so to sum up the string vs int debate:
>> > > > > > >> >> >>
>> > > > > > >> >> >> performance - you can do 140k ops/sec _per thread_ with
>> > > string
>> > > > > > >> headers.
>> > > > > > >> >> you
>> > > > > > >> >> >> could do x2-3 better with ints. there's no arguing the
>> > > > relative
>> > > > > > diff
>> > > > > > >> >> >> between the two, there's only the question of whether or
>> > not
>> > > > > _the
>> > > > > > >> rest
>> > > > > > >> >> of
>> > > > > > >> >> >> kafka_ operates fast enough to care. if we want to make
>> > > > choices
>> > > > > > >> solely
>> > > > > > >> >> >> based on performance we need ints. if we are willing to
>> > > > > > >> >> settle/compromise
>> > > > > > >> >> >> for a nicer (to some) API than strings are good enough
>> for
>> > > the
>> > > > > > >> current
>> > > > > > >> >> >> state of affairs.
>> > > > > > >> >> >>
>> > > > > > >> >> >> message size - with batching and compression it comes
>> down
>> > > to
>> > > > a
>> > > > > > ~5%
>> > > > > > >> >> >> difference (internal testing, not in the doc. maybe
>> would
>> > > help
>> > > > > > >> adding if
>> > > > > > >> >> >> this becomes a point of contention?). this means it wont
>> > > > really
>> > > > > > >> affect
>> > > > > > >> >> >> kafka in "throughput mode" (large, compressed batches).
>> in
>> > > > "low
>> > > > > > >> latency"
>> > > > > > >> >> >> mode (meaning less/no batching and compression) the
>> > > difference
>> > > > > can
>> > > > > > >> be
>> > > > > > >> >> >> extreme (it'll easily be an order of magnitude with
>> small
>> > > > > payloads
>> > > > > > >> like
>> > > > > > >> >> >> stock ticks and header keys of the form
>> > > > > > >> >> >> "com.acme.infraTeam.kafka.hiMom.auditPlugin"). we have
>> a
>> > > few
>> > > > > such
>> > > > > > >> >> topics at
>> > > > > > >> >> >> linkedin where actual payloads are ~2 ints and are
>> > eclipsed
>> > > by
>> > > > > our
>> > > > > > >> >> in-house
>> > > > > > >> >> >> audit "header" which is why we liked ints to begin with.
>> > > > > > >> >> >>
>> > > > > > >> >> >> "ease of use" - strings would probably still require
>> > _some_
>> > > > > degree
>> > > > > > >> of
>> > > > > > >> >> >> partitioning by convention (imagine if everyone used the
>> > key
>> > > > > > >> "infra"...)
>> > > > > > >> >> >> but its very intuitive for java devs to do anyway
>> > > > > (reverse-domain
>> > > > > > is
>> > > > > > >> >> >> ingrained into java developers at a young age :-) ).
>> also
>> > > most
>> > > > > > java
>> > > > > > >> devs
>> > > > > > >> >> >> find Map<String, whatever> more intuitive than
>> > Map<Integer,
>> > > > > > >> whatever> -
>> > > > > > >> >> >> probably because of other text-based protocols like
>> http.
>> > > ints
>> > > > > > would
>> > > > > > >> >> >> require a number registry. if you think number
>> registries
>> > > are
>> > > > > hard
>> > > > > > >> just
>> > > > > > >> >> >> look at the wiki page for KIPs (specifically the number
>> > for
>> > > > next
>> > > > > > >> >> available
>> > > > > > >> >> >> KIP) and think again - we are probably talking about the
>> > > same
>> > > > > > >> volume of
>> > > > > > >> >> >> requests. also this would only be "required" (good
>> > > > citizenship,
>> > > > > > more
>> > > > > > >> >> like)
>> > > > > > >> >> >> if you want to publish your plugin for others to use.
>> > within
>> > > > > your
>> > > > > > >> org do
>> > > > > > >> >> >> whatever you want - just know that if you use [some
>> > > "reserved"
>> > > > > > >> range]
>> > > > > > >> >> and a
>> > > > > > >> >> >> future kafka update breaks it its your problem. RTFM.
>> > > > > > >> >> >>
>> > > > > > >> >> >> personally im in favor of ints.
>> > > > > > >> >> >>
>> > > > > > >> >> >> having said that (and like nacho) I will settle if int
>> vs
>> > > > string
>> > > > > > >> remains
>> > > > > > >> >> >> the only obstacle to this.
>> > > > > > >> >> >>
>> > > > > > >> >> >> On Tue, Nov 8, 2016 at 3:53 PM, Nacho Solis
>> > > > > > >> <nso...@linkedin.com.invalid
>> > > > > > >> >> >
>> > > > > > >> >> >> wrote:
>> > > > > > >> >> >>
>> > > > > > >> >> >>> I think it's well known I've been pushing for ints
>> (and I
>> > > > could
>> > > > > > >> switch
>> > > > > > >> >> to
>> > > > > > >> >> >>> 16 bit shorts if pressed).
>> > > > > > >> >> >>>
>> > > > > > >> >> >>> - efficient (space)
>> > > > > > >> >> >>> - efficient (processing)
>> > > > > > >> >> >>> - easily partitionable
>> > > > > > >> >> >>>
>> > > > > > >> >> >>>
>> > > > > > >> >> >>> However, if the only thing that is keeping us from
>> > adopting
>> > > > > > >> headers is
>> > > > > > >> >> the
>> > > > > > >> >> >>> use of strings vs ints as keys, then I would cave in
>> and
>> > > > accept
>> > > > > > >> >> strings. If
>> > > > > > >> >> >>> we do so, I would like to limit string keys to 128
>> bytes
>> > in
>> > > > > > length.
>> > > > > > >> >> This
>> > > > > > >> >> >>> way 1) I could use a 3 letter string if I wanted
>> > > (effectively
>> > > > > > >> using 4
>> > > > > > >> >> total
>> > > > > > >> >> >>> bytes), 2) limit overall impact of possible keys (don't
>> > > > really
>> > > > > > want
>> > > > > > >> >> people
>> > > > > > >> >> >>> to send a 16K header string key).
>> > > > > > >> >> >>>
>> > > > > > >> >> >>> Nacho
>> > > > > > >> >> >>>
>> > > > > > >> >> >>>
>> > > > > > >> >> >>> On Tue, Nov 8, 2016 at 3:35 PM, Gwen Shapira <
>> > > > > g...@confluent.io>
>> > > > > > >> >> wrote:
>> > > > > > >> >> >>>
>> > > > > > >> >> >>>> Forgot to mention: Thank you for quantifying the
>> > > trade-off -
>> > > > > it
>> > > > > > is
>> > > > > > >> >> >>>> helpful and important regardless of what we end up
>> > > deciding.
>> > > > > > >> >> >>>>
>> > > > > > >> >> >>>> On Tue, Nov 8, 2016 at 3:12 PM, Sean McCauliff
>> > > > > > >> >> >>>> <smccaul...@linkedin.com.invalid> wrote:
>> > > > > > >> >> >>>>> On Tue, Nov 8, 2016 at 2:15 PM, Gwen Shapira <
>> > > > > > g...@confluent.io>
>> > > > > > >> >> >>> wrote:
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>> Since Kafka specifically targets high-throughput,
>> > > > > low-latency
>> > > > > > >> >> >>>>>> use-cases, I don't think we should trade them off
>> that
>> > > > > easily.
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>> I find these kind of design goals not to be really
>> > > helpful
>> > > > > > unless
>> > > > > > >> >> it's
>> > > > > > >> >> >>>>> quantified in someway. Because it's always possible
>> to
>> > > > argue
>> > > > > > >> against
>> > > > > > >> >> >>>>> something as either being not performant or just an
>> > > > > > >> implementation
>> > > > > > >> >> >>>> detail.
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>> This is a single threaded benchmarks so all the
>> > > > measurements
>> > > > > > are
>> > > > > > >> per
>> > > > > > >> >> >>>>> thread.
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>> For 1M messages/s/thread if header keys are int and
>> > you
>> > > > had
>> > > > > > >> even a
>> > > > > > >> >> >>>> single
>> > > > > > >> >> >>>>> header key, value pair then it's still about 2^-2
>> > > > > microseconds
>> > > > > > >> which
>> > > > > > >> >> >>>> means
>> > > > > > >> >> >>>>> you only have another 0.75 microseconds to do
>> > everything
>> > > > else
>> > > > > > you
>> > > > > > >> >> want
>> > > > > > >> >> >>> to
>> > > > > > >> >> >>>>> do with a message (1M messages/s means 1 micro second
>> > per
>> > > > > > >> message).
>> > > > > > >> >> >>> With
>> > > > > > >> >> >>>>> string header keys there is still 0.5 micro seconds
>> to
>> > > > > process
>> > > > > > a
>> > > > > > >> >> >>> message.
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>> I love strings as much as the next guy (we had them
>> in
>> > > > > Flume),
>> > > > > > >> but I
>> > > > > > >> >> >>>>>> was convinced by Magnus/Michael/Radai that strings
>> > don't
>> > > > > > >> actually
>> > > > > > >> >> have
>> > > > > > >> >> >>>>>> strong benefits as opposed to ints (you'll need a
>> > string
>> > > > > > >> registry
>> > > > > > >> >> >>>>>> anyway - otherwise, how will you know what does the
>> > > > > > "profile_id"
>> > > > > > >> >> >>>>>> header refers to?) and I want to keep closer to our
>> > > > original
>> > > > > > >> design
>> > > > > > >> >> >>>>>> goals for Kafka.
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>> "confluent.profile_id"
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>> If someone likes strings in the headers and doesn't
>> do
>> > > > > > millions
>> > > > > > >> of
>> > > > > > >> >> >>>>>> messages a sec, they probably have lots of other
>> > systems
>> > > > > they
>> > > > > > >> can
>> > > > > > >> >> use
>> > > > > > >> >> >>>>>> instead.
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>> None of them will scale like Kafka. Horizontal
>> scaling
>> > > is
>> > > > > > still
>> > > > > > >> >> good.
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>> On Tue, Nov 8, 2016 at 1:22 PM, Sean McCauliff
>> > > > > > >> >> >>>>>> <smccaul...@linkedin.com.invalid> wrote:
>> > > > > > >> >> >>>>>>> +1 for String keys.
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>> I've been doing some bechmarking and it seems like
>> > the
>> > > > > > speedup
>> > > > > > >> for
>> > > > > > >> >> >>>> using
>> > > > > > >> >> >>>>>>> integer keys is about 2-5 depending on the length
>> of
>> > > the
>> > > > > > >> strings
>> > > > > > >> >> and
>> > > > > > >> >> >>>> what
>> > > > > > >> >> >>>>>>> collections are being used. The overall amount of
>> > time
>> > > > > spent
>> > > > > > >> >> >>> parsing
>> > > > > > >> >> >>>> a
>> > > > > > >> >> >>>>>> set
>> > > > > > >> >> >>>>>>> of header key, value pairs probably does not matter
>> > > > unless
>> > > > > > you
>> > > > > > >> are
>> > > > > > >> >> >>>>>> getting
>> > > > > > >> >> >>>>>>> close to 1M messages per consumer. In which case
>> > > > probably
>> > > > > > >> don't
>> > > > > > >> >> use
>> > > > > > >> >> >>>>>>> headers. There is also the option to use very
>> short
>> > > > > strings;
>> > > > > > >> some
>> > > > > > >> >> >>>> that
>> > > > > > >> >> >>>>>> are
>> > > > > > >> >> >>>>>>> even shorter than integers.
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>> Partitioning the string key space will be easier
>> than
>> > > > > > >> partitioning
>> > > > > > >> >> >>> an
>> > > > > > >> >> >>>>>>> integer key space. We won't need a global registry.
>> > > > Kafka
>> > > > > > >> >> >>> internally
>> > > > > > >> >> >>>> can
>> > > > > > >> >> >>>>>>> reserve some prefix like "_" as its namespace.
>> > > Everyone
>> > > > > else
>> > > > > > >> can
>> > > > > > >> >> >>> use
>> > > > > > >> >> >>>>>> their
>> > > > > > >> >> >>>>>>> company or project name as namespace prefix and
>> life
>> > > > should
>> > > > > > be
>> > > > > > >> >> good.
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>> Here's the link to some of the benchmarking info:
>> > > > > > >> >> >>>>>>> https://docs.google.com/document/d/1tfT-
>> > > > > > >> >> >>>> 6SZdnKOLyWGDH82kS30PnUkmgb7nPL
>> > > > > > >> >> >>>>>> dw6p65pAI/edit?usp=sharing
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>> --
>> > > > > > >> >> >>>>>>> Sean McCauliff
>> > > > > > >> >> >>>>>>> Staff Software Engineer
>> > > > > > >> >> >>>>>>> Kafka
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>> smccaul...@linkedin.com
>> > > > > > >> >> >>>>>>> linkedin.com/in/sean-mccauliff-b563192
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>> On Mon, Nov 7, 2016 at 11:51 PM, Michael Pearce <
>> > > > > > >> >> >>>> michael.pea...@ig.com>
>> > > > > > >> >> >>>>>>> wrote:
>> > > > > > >> >> >>>>>>>
>> > > > > > >> >> >>>>>>>> +1 on this slimmer version of our proposal
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> I def think the Id space we can reduce from the
>> > > proposed
>> > > > > > >> >> >>>> int32(4bytes)
>> > > > > > >> >> >>>>>>>> down to int16(2bytes) it saves on space and as
>> > headers
>> > > > we
>> > > > > > >> wouldn't
>> > > > > > >> >> >>>>>> expect
>> > > > > > >> >> >>>>>>>> the number of headers being used concurrently
>> being
>> > > that
>> > > > > > high.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> I would wonder if we should make the value byte
>> > array
>> > > > > length
>> > > > > > >> still
>> > > > > > >> >> >>>> int32
>> > > > > > >> >> >>>>>>>> though as This is the standard Max array length in
>> > > Java
>> > > > > > saying
>> > > > > > >> >> that
>> > > > > > >> >> >>>> it
>> > > > > > >> >> >>>>>> is a
>> > > > > > >> >> >>>>>>>> header and I guess limiting the size is sensible
>> and
>> > > > would
>> > > > > > >> work
>> > > > > > >> >> for
>> > > > > > >> >> >>>> all
>> > > > > > >> >> >>>>>> the
>> > > > > > >> >> >>>>>>>> use cases we have in mind so happy with limiting
>> > this.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Do people generally concur on Magnus's slimmer
>> > > version?
>> > > > > > >> Anyone see
>> > > > > > >> >> >>>> any
>> > > > > > >> >> >>>>>>>> issues if we moved from int32 to int16?
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Re configurable ids per plugin over a global
>> > registry
>> > > > also
>> > > > > > >> would
>> > > > > > >> >> >>> work
>> > > > > > >> >> >>>>>> for
>> > > > > > >> >> >>>>>>>> us. As such if this has better concensus over the
>> > > > > proposed
>> > > > > > >> global
>> > > > > > >> >> >>>>>> registry
>> > > > > > >> >> >>>>>>>> I'd be happy to change that.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> I was already sold on ints over strings for keys
>> ;)
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Cheers
>> > > > > > >> >> >>>>>>>> Mike
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> ________________________________________
>> > > > > > >> >> >>>>>>>> From: Magnus Edenhill <mag...@edenhill.se>
>> > > > > > >> >> >>>>>>>> Sent: Monday, November 7, 2016 10:10:21 PM
>> > > > > > >> >> >>>>>>>> To: dev@kafka.apache.org
>> > > > > > >> >> >>>>>>>> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Hi,
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> I'm +1 for adding generic message headers, but I
>> do
>> > > > share
>> > > > > > the
>> > > > > > >> >> >>>> concerns
>> > > > > > >> >> >>>>>>>> previously aired on this thread and during the KIP
>> > > > > meeting.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> So let me propose a slimmer alternative that does
>> > not
>> > > > > > require
>> > > > > > >> any
>> > > > > > >> >> >>>> sort
>> > > > > > >> >> >>>>>> of
>> > > > > > >> >> >>>>>>>> global header registry, does not affect broker
>> > > > performance
>> > > > > > or
>> > > > > > >> >> >>>>>> operations,
>> > > > > > >> >> >>>>>>>> and adds as little overhead as possible.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Message
>> > > > > > >> >> >>>>>>>> ------------
>> > > > > > >> >> >>>>>>>> The protocol Message type is extended with a
>> Headers
>> > > > array
>> > > > > > >> >> consting
>> > > > > > >> >> >>>> of
>> > > > > > >> >> >>>>>>>> Tags, where a Tag is defined as:
>> > > > > > >> >> >>>>>>>> int16 Id
>> > > > > > >> >> >>>>>>>> int16 Len // binary_data length
>> > > > > > >> >> >>>>>>>> binary_data[Len] // opaque binary data
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Ids
>> > > > > > >> >> >>>>>>>> ---
>> > > > > > >> >> >>>>>>>> The Id space is not centrally managed, so whenever
>> > an
>> > > > > > >> application
>> > > > > > >> >> >>>> needs
>> > > > > > >> >> >>>>>> to
>> > > > > > >> >> >>>>>>>> add headers, or use an eco-system plugin that
>> does,
>> > > its
>> > > > Id
>> > > > > > >> >> >>> allocation
>> > > > > > >> >> >>>>>> will
>> > > > > > >> >> >>>>>>>> need to be manually configured.
>> > > > > > >> >> >>>>>>>> This moves the allocation concern from the global
>> > > space
>> > > > > down
>> > > > > > >> to
>> > > > > > >> >> >>>>>>>> organization level and avoids the risk for id
>> > > conflicts.
>> > > > > > >> >> >>>>>>>> Example pseudo-config for some app:
>> > > > > > >> >> >>>>>>>> sometrackerplugin.tag.sourcev3.id=1000
>> > > > > > >> >> >>>>>>>> dbthing.tag.tablename.id=1001
>> > > > > > >> >> >>>>>>>> myschemareg.tag.schemaname.id=1002
>> > > > > > >> >> >>>>>>>> myschemareg.tag.schemaversion.id=1003
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Each header-writing or header-reading plugin must
>> > > > provide
>> > > > > > >> means
>> > > > > > >> >> >>>>>> (typically
>> > > > > > >> >> >>>>>>>> through configuration) to specify the tag for each
>> > > > header
>> > > > > it
>> > > > > > >> uses.
>> > > > > > >> >> >>>>>> Defaults
>> > > > > > >> >> >>>>>>>> should be avoided.
>> > > > > > >> >> >>>>>>>> A consumer silently ignores tags it does not have
>> a
>> > > > > mapping
>> > > > > > >> for
>> > > > > > >> >> >>>> (since
>> > > > > > >> >> >>>>>> the
>> > > > > > >> >> >>>>>>>> binary_data can't be parsed without knowing what
>> it
>> > > is).
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Id range 0..999 is reserved for future use by the
>> > > broker
>> > > > > and
>> > > > > > >> must
>> > > > > > >> >> >>>> not be
>> > > > > > >> >> >>>>>>>> used by plugins.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Broker
>> > > > > > >> >> >>>>>>>> ---------
>> > > > > > >> >> >>>>>>>> The broker does not process the tags (other than
>> the
>> > > > > > standard
>> > > > > > >> >> >>>> protocol
>> > > > > > >> >> >>>>>>>> syntax verification), it simply stores and
>> forwards
>> > > them
>> > > > > as
>> > > > > > >> opaque
>> > > > > > >> >> >>>> data.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Standard message translation (removal of Headers)
>> > > kicks
>> > > > in
>> > > > > > for
>> > > > > > >> >> >>> older
>> > > > > > >> >> >>>>>>>> clients.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Why not string ids?
>> > > > > > >> >> >>>>>>>> -------------------------
>> > > > > > >> >> >>>>>>>> String ids might seem like a good idea, but:
>> > > > > > >> >> >>>>>>>> * does not really solve uniqueness
>> > > > > > >> >> >>>>>>>> * consumes a lot of space (2 byte string length +
>> > > > string,
>> > > > > > per
>> > > > > > >> >> >>>> header)
>> > > > > > >> >> >>>>>> to
>> > > > > > >> >> >>>>>>>> be meaningful
>> > > > > > >> >> >>>>>>>> * doesn't really say anything how to parse the
>> tag's
>> > > > data,
>> > > > > > so
>> > > > > > >> it
>> > > > > > >> >> >>> is
>> > > > > > >> >> >>>> in
>> > > > > > >> >> >>>>>>>> effect useless on its own.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> Regards,
>> > > > > > >> >> >>>>>>>> Magnus
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>> 2016-11-07 18:32 GMT+01:00 Michael Pearce <
>> > > > > > >> michael.pea...@ig.com
>> > > > > > >> >> >:
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>>>> Hi Roger,
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> Thanks for the support.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> I think the key thing is to have a common key
>> space
>> > > to
>> > > > > make
>> > > > > > >> an
>> > > > > > >> >> >>>>>> ecosystem,
>> > > > > > >> >> >>>>>>>>> there does have to be some level of contract for
>> > > people
>> > > > > to
>> > > > > > >> play
>> > > > > > >> >> >>>>>> nicely.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> Having map<String, byte[]> or as per current
>> > proposed
>> > > > in
>> > > > > > kip
>> > > > > > >> of
>> > > > > > >> >> >>>>>> having a
>> > > > > > >> >> >>>>>>>>> numerical key space of map<int, byte[]> is a
>> level
>> > > of
>> > > > > the
>> > > > > > >> >> >>> contract
>> > > > > > >> >> >>>>>> that
>> > > > > > >> >> >>>>>>>>> most people would expect.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> I think the example in a previous comment someone
>> > > else
>> > > > > made
>> > > > > > >> >> >>>> linking to
>> > > > > > >> >> >>>>>>>> AWS
>> > > > > > >> >> >>>>>>>>> blog and also implemented api where originally
>> they
>> > > > > didn't
>> > > > > > >> have a
>> > > > > > >> >> >>>>>> header
>> > > > > > >> >> >>>>>>>>> space but not they do, where keys are uniform but
>> > the
>> > > > > value
>> > > > > > >> can
>> > > > > > >> >> >>> be
>> > > > > > >> >> >>>>>>>> string,
>> > > > > > >> >> >>>>>>>>> int, anything is a good example.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> Having a custom MetadataSerializer is something
>> we
>> > > had
>> > > > > > played
>> > > > > > >> >> >>> with,
>> > > > > > >> >> >>>>>> but
>> > > > > > >> >> >>>>>>>>> discounted the idea, as if you wanted everyone to
>> > > work
>> > > > > the
>> > > > > > >> same
>> > > > > > >> >> >>>> way in
>> > > > > > >> >> >>>>>>>> the
>> > > > > > >> >> >>>>>>>>> ecosystem, having to have this also customizable
>> > > makes
>> > > > > it a
>> > > > > > >> bit
>> > > > > > >> >> >>>>>> harder.
>> > > > > > >> >> >>>>>>>>> Think about making the whole message record
>> custom
>> > > > > > >> serializable,
>> > > > > > >> >> >>>> this
>> > > > > > >> >> >>>>>>>> would
>> > > > > > >> >> >>>>>>>>> make it fairly tricky (though it would not be
>> > > > impossible)
>> > > > > > to
>> > > > > > >> have
>> > > > > > >> >> >>>> made
>> > > > > > >> >> >>>>>>>> work
>> > > > > > >> >> >>>>>>>>> nicely. Having the value customizable we thought
>> > is a
>> > > > > > >> reasonable
>> > > > > > >> >> >>>>>> tradeoff
>> > > > > > >> >> >>>>>>>>> here of flexibility over contract of interaction
>> > > > between
>> > > > > > >> >> >>> different
>> > > > > > >> >> >>>>>>>> parties.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> Is there a particular case or benefit of having
>> > > > > > serialization
>> > > > > > >> >> >>>>>>>> customizable
>> > > > > > >> >> >>>>>>>>> that you have in mind?
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> Saying this it is obviously something that could
>> be
>> > > > > > >> implemented,
>> > > > > > >> >> >>> if
>> > > > > > >> >> >>>>>> there
>> > > > > > >> >> >>>>>>>>> is a need. If we did go this avenue I think a
>> > > defaulted
>> > > > > > >> >> >>> serializer
>> > > > > > >> >> >>>>>>>>> implementation should exist so for the 80:20
>> rule,
>> > > > people
>> > > > > > can
>> > > > > > >> >> >>> just
>> > > > > > >> >> >>>>>> have
>> > > > > > >> >> >>>>>>>> the
>> > > > > > >> >> >>>>>>>>> broker and clients get default behavior.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> Cheers
>> > > > > > >> >> >>>>>>>>> Mike
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> On 11/6/16, 5:25 PM, "radai" <
>> > > > radai.rosenbl...@gmail.com
>> > > > > >
>> > > > > > >> wrote:
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> making header _key_ serialization configurable
>> > > > > > potentially
>> > > > > > >> >> >>>>>> undermines
>> > > > > > >> >> >>>>>>>>> the
>> > > > > > >> >> >>>>>>>>> board usefulness of the feature (any point
>> along
>> > > the
>> > > > > > path
>> > > > > > >> >> >>> must
>> > > > > > >> >> >>>> be
>> > > > > > >> >> >>>>>>>> able
>> > > > > > >> >> >>>>>>>>> to
>> > > > > > >> >> >>>>>>>>> read the header keys. the values may be
>> whatever
>> > > and
>> > > > > > >> require
>> > > > > > >> >> >>>> more
>> > > > > > >> >> >>>>>>>>> intimate
>> > > > > > >> >> >>>>>>>>> knowledge of the code that produced specific
>> > > > headers,
>> > > > > > but
>> > > > > > >> >> >>> keys
>> > > > > > >> >> >>>>>> should
>> > > > > > >> >> >>>>>>>>> be
>> > > > > > >> >> >>>>>>>>> universally readable).
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> it would also make it hard to write really
>> > > portable
>> > > > > > >> plugins -
>> > > > > > >> >> >>>> say
>> > > > > > >> >> >>>>>> i
>> > > > > > >> >> >>>>>>>>> wrote a
>> > > > > > >> >> >>>>>>>>> large message splitter/combiner - if i rely on
>> > key
>> > > > > > >> >> >>>> "largeMessage"
>> > > > > > >> >> >>>>>> and
>> > > > > > >> >> >>>>>>>>> values of the form "1/20" someone who uses
>> > > > (contrived
>> > > > > > >> >> >>> example)
>> > > > > > >> >> >>>>>>>>> Map<Byte[],
>> > > > > > >> >> >>>>>>>>> Double> wouldnt be able to re-use my code.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> not the end of a the world within an
>> > organization,
>> > > > but
>> > > > > > >> >> >>>>>> problematic if
>> > > > > > >> >> >>>>>>>>> you
>> > > > > > >> >> >>>>>>>>> want to enable an ecosystem
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> On Thu, Nov 3, 2016 at 2:04 PM, Roger Hoover <
>> > > > > > >> >> >>>>>> roger.hoo...@gmail.com
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> wrote:
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> As others have laid out, I see strong reasons
>> for
>> > a
>> > > > > common
>> > > > > > >> >> >>>>>> message
>> > > > > > >> >> >>>>>>>>>> metadata structure for the Kafka ecosystem. In
>> > > > > > particular,
>> > > > > > >> >> >>>> I've
>> > > > > > >> >> >>>>>>>>> seen that
>> > > > > > >> >> >>>>>>>>>> even within a single organization,
>> infrastructure
>> > > > teams
>> > > > > > >> >> >>> often
>> > > > > > >> >> >>>>>> own
>> > > > > > >> >> >>>>>>>> the
>> > > > > > >> >> >>>>>>>>>> message metadata while application teams own the
>> > > > > > >> >> >>>>>> application-level
>> > > > > > >> >> >>>>>>>>> data
>> > > > > > >> >> >>>>>>>>>> format. Allowing metadata and content to have
>> > > > different
>> > > > > > >> >> >>>>>> structure
>> > > > > > >> >> >>>>>>>>> and
>> > > > > > >> >> >>>>>>>>>> evolve separately is very helpful for this.
>> > Also, I
>> > > > > think
>> > > > > > >> >> >>>>>> there's
>> > > > > > >> >> >>>>>>>> a
>> > > > > > >> >> >>>>>>>>> lot of
>> > > > > > >> >> >>>>>>>>>> value to having a common metadata structure
>> shared
>> > > > > across
>> > > > > > >> >> >>> the
>> > > > > > >> >> >>>>>> Kafka
>> > > > > > >> >> >>>>>>>>>> ecosystem so that tools which leverage metadata
>> > can
>> > > > more
>> > > > > > >> >> >>>> easily
>> > > > > > >> >> >>>>>> be
>> > > > > > >> >> >>>>>>>>> shared
>> > > > > > >> >> >>>>>>>>>> across organizations and integrated together.
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> The question is, where does the metadata
>> structure
>> > > > > belong?
>> > > > > > >> >> >>>>>> Here's
>> > > > > > >> >> >>>>>>>>> my take:
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> We change the Kafka wire and on-disk format to
>> > from
>> > > a
>> > > > > > (key,
>> > > > > > >> >> >>>>>> value)
>> > > > > > >> >> >>>>>>>>> model to
>> > > > > > >> >> >>>>>>>>>> a (key, metadata, value) model where all three
>> are
>> > > > byte
>> > > > > > >> >> >>>> arrays
>> > > > > > >> >> >>>>>> from
>> > > > > > >> >> >>>>>>>>> the
>> > > > > > >> >> >>>>>>>>>> brokers point of view. The primary reason for
>> > this
>> > > is
>> > > > > > that
>> > > > > > >> >> >>>> it
>> > > > > > >> >> >>>>>>>>> provides a
>> > > > > > >> >> >>>>>>>>>> backward compatible migration path forward.
>> > > Producers
>> > > > > can
>> > > > > > >> >> >>>> start
>> > > > > > >> >> >>>>>>>>> populating
>> > > > > > >> >> >>>>>>>>>> metadata fields before all consumers understand
>> > the
>> > > > > > >> >> >>> metadata
>> > > > > > >> >> >>>>>>>>> structure.
>> > > > > > >> >> >>>>>>>>>> For people who already have custom envelope
>> > > > structures,
>> > > > > > >> >> >>> they
>> > > > > > >> >> >>>> can
>> > > > > > >> >> >>>>>>>>> populate
>> > > > > > >> >> >>>>>>>>>> their existing structure and the new structure
>> > for a
>> > > > > while
>> > > > > > >> >> >>> as
>> > > > > > >> >> >>>>>> they
>> > > > > > >> >> >>>>>>>>> make the
>> > > > > > >> >> >>>>>>>>>> transition.
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> We could stop there and let the clients plug in
>> a
>> > > > > > >> >> >>>> KeySerializer,
>> > > > > > >> >> >>>>>>>>>> MetadataSerializer, and ValueSerializer but I
>> > think
>> > > it
>> > > > > is
>> > > > > > >> >> >>>> also
>> > > > > > >> >> >>>>>> be
>> > > > > > >> >> >>>>>>>>> useful to
>> > > > > > >> >> >>>>>>>>>> have a default MetadataSerializer that
>> implements
>> > a
>> > > > > > >> >> >>> key-value
>> > > > > > >> >> >>>>>> model
>> > > > > > >> >> >>>>>>>>> similar
>> > > > > > >> >> >>>>>>>>>> to AMQP or HTTP headers. Or we could go even
>> > > further
>> > > > > and
>> > > > > > >> >> >>>>>>>> prescribe a
>> > > > > > >> >> >>>>>>>>>> Map<String, byte[]> or Map<String, String> data
>> > > model
>> > > > > for
>> > > > > > >> >> >>>>>> headers
>> > > > > > >> >> >>>>>>>> in
>> > > > > > >> >> >>>>>>>>> the
>> > > > > > >> >> >>>>>>>>>> clients (while still allowing custom
>> serialization
>> > > of
>> > > > > the
>> > > > > > >> >> >>>> header
>> > > > > > >> >> >>>>>>>> data
>> > > > > > >> >> >>>>>>>>>> model).
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> I think this would address Radai's concerns:
>> > > > > > >> >> >>>>>>>>>> 1. All client code would not need to be updated
>> to
>> > > > know
>> > > > > > >> >> >>> about
>> > > > > > >> >> >>>>>> the
>> > > > > > >> >> >>>>>>>>>> container.
>> > > > > > >> >> >>>>>>>>>> 2. Middleware friendly clients would have a
>> > standard
>> > > > > > header
>> > > > > > >> >> >>>> data
>> > > > > > >> >> >>>>>>>>> model to
>> > > > > > >> >> >>>>>>>>>> work with.
>> > > > > > >> >> >>>>>>>>>> 3. KIP is required both b/c of broker changes
>> and
>> > > > > because
>> > > > > > >> >> >>> of
>> > > > > > >> >> >>>>>> client
>> > > > > > >> >> >>>>>>>>> API
>> > > > > > >> >> >>>>>>>>>> changes.
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> Cheers,
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> Roger
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>> On Wed, Nov 2, 2016 at 4:38 PM, radai <
>> > > > > > >> >> >>>>>> radai.rosenbl...@gmail.com>
>> > > > > > >> >> >>>>>>>>> wrote:
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>>> my biggest issues with a "standard" wrapper
>> > format:
>> > > > > > >> >> >>>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>>> 1. _ALL_ client _CODE_ (as opposed to kafka lib
>> > > > > version)
>> > > > > > >> >> >>>> must
>> > > > > > >> >> >>>>>> be
>> > > > > > >> >> >>>>>>>>> updated
>> > > > > > >> >> >>>>>>>>>> to
>> > > > > > >> >> >>>>>>>>>>> know about the container, because any old naive
>> > > code
>> > > > > > >> >> >>>> trying to
>> > > > > > >> >> >>>>>>>>> directly
>> > > > > > >> >> >>>>>>>>>>> deserialize its own payload would keel over and
>> > die
>> > > > (it
>> > > > > > >> >> >>>> needs
>> > > > > > >> >> >>>>>> to
>> > > > > > >> >> >>>>>>>>> know to
>> > > > > > >> >> >>>>>>>>>>> deserialize a container, and then dig in there
>> > for
>> > > > its
>> > > > > > >> >> >>>>>> payload).
>> > > > > > >> >> >>>>>>>>>>> 2. in order to write middleware-friendly
>> clients
>> > > that
>> > > > > > >> >> >>>> utilize
>> > > > > > >> >> >>>>>>>> such
>> > > > > > >> >> >>>>>>>>> a
>> > > > > > >> >> >>>>>>>>>>> container one would basically have to write
>> their
>> > > own
>> > > > > > >> >> >>>>>>>>> producer/consumer
>> > > > > > >> >> >>>>>>>>>> API
>> > > > > > >> >> >>>>>>>>>>> on top of the open source kafka one.
>> > > > > > >> >> >>>>>>>>>>> 3. if you were going to go with a wrapper
>> format
>> > > you
>> > > > > > >> >> >>> really
>> > > > > > >> >> >>>>>> dont
>> > > > > > >> >> >>>>>>>>> need to
>> > > > > > >> >> >>>>>>>>>>> bother with a kip (just open source your own
>> > client
>> > > > > stack
>> > > > > > >> >> >>>>>> from #2
>> > > > > > >> >> >>>>>>>>> above
>> > > > > > >> >> >>>>>>>>>> so
>> > > > > > >> >> >>>>>>>>>>> others could stop re-inventing it)
>> > > > > > >> >> >>>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>>> On Wed, Nov 2, 2016 at 4:25 PM, James Cheng <
>> > > > > > >> >> >>>>>>>> wushuja...@gmail.com>
>> > > > > > >> >> >>>>>>>>>> wrote:
>> > > > > > >> >> >>>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>>>> How exactly would this work? Or maybe that's
>> out
>> > > of
>> > > > > > >> >> >>> scope
>> > > > > > >> >> >>>>>> for
>> > > > > > >> >> >>>>>>>>> this
>> > > > > > >> >> >>>>>>>>>> email.
>> > > > > > >> >> >>>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>>
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>>> The information contained in this email is
>> strictly
>> > > > > > >> confidential
>> > > > > > >> >> >>>> and
>> > > > > > >> >> >>>>>> for
>> > > > > > >> >> >>>>>>>>> the use of the addressee only, unless otherwise
>> > > > > indicated.
>> > > > > > >> If you
>> > > > > > >> >> >>>> are
>> > > > > > >> >> >>>>>> not
>> > > > > > >> >> >>>>>>>>> the intended recipient, please do not read, copy,
>> > use
>> > > > or
>> > > > > > >> disclose
>> > > > > > >> >> >>>> to
>> > > > > > >> >> >>>>>>>> others
>> > > > > > >> >> >>>>>>>>> this message or any attachment. Please also
>> notify
>> > > the
>> > > > > > >> sender by
>> > > > > > >> >> >>>>>> replying
>> > > > > > >> >> >>>>>>>>> to this email or by telephone (+44(020 7896 0011)
>> > and
>> > > > > then
>> > > > > > >> delete
>> > > > > > >> >> >>>> the
>> > > > > > >> >> >>>>>>>> email
>> > > > > > >> >> >>>>>>>>> and any copies of it. Opinions, conclusion (etc)
>> > that
>> > > > do
>> > > > > > not
>> > > > > > >> >> >>>> relate to
>> > > > > > >> >> >>>>>>>> the
>> > > > > > >> >> >>>>>>>>> official business of this company shall be
>> > understood
>> > > > as
>> > > > > > >> neither
>> > > > > > >> >> >>>> given
>> > > > > > >> >> >>>>>>>> nor
>> > > > > > >> >> >>>>>>>>> endorsed by it. IG is a trading name of IG
>> Markets
>> > > > > Limited
>> > > > > > (a
>> > > > > > >> >> >>>> company
>> > > > > > >> >> >>>>>>>>> registered in England and Wales, company number
>> > > > 04008957)
>> > > > > > >> and IG
>> > > > > > >> >> >>>> Index
>> > > > > > >> >> >>>>>>>>> Limited (a company registered in England and
>> Wales,
>> > > > > company
>> > > > > > >> >> >>> number
>> > > > > > >> >> >>>>>>>>> 01190902). Registered address at Cannon Bridge
>> > House,
>> > > > 25
>> > > > > > >> Dowgate
>> > > > > > >> >> >>>> Hill,
>> > > > > > >> >> >>>>>>>>> London EC4R 2YA. Both IG Markets Limited
>> (register
>> > > > number
>> > > > > > >> 195355)
>> > > > > > >> >> >>>> and
>> > > > > > >> >> >>>>>> IG
>> > > > > > >> >> >>>>>>>>> Index Limited (register number 114059) are
>> > authorised
>> > > > and
>> > > > > > >> >> >>>> regulated by
>> > > > > > >> >> >>>>>>>> the
>> > > > > > >> >> >>>>>>>>> Financial Conduct Authority.
>> > > > > > >> >> >>>>>>>>>
>> > > > > > >> >> >>>>>>>> The information contained in this email is
>> strictly
>> > > > > > >> confidential
>> > > > > > >> >> >>> and
>> > > > > > >> >> >>>> for
>> > > > > > >> >> >>>>>>>> the use of the addressee only, unless otherwise
>> > > > indicated.
>> > > > > > If
>> > > > > > >> you
>> > > > > > >> >> >>> are
>> > > > > > >> >> >>>>>> not
>> > > > > > >> >> >>>>>>>> the intended recipient, please do not read, copy,
>> > use
>> > > or
>> > > > > > >> disclose
>> > > > > > >> >> >>> to
>> > > > > > >> >> >>>>>> others
>> > > > > > >> >> >>>>>>>> this message or any attachment. Please also notify
>> > the
>> > > > > > sender
>> > > > > > >> by
>> > > > > > >> >> >>>>>> replying
>> > > > > > >> >> >>>>>>>> to this email or by telephone (+44(020 7896 0011)
>> > and
>> > > > then
>> > > > > > >> delete
>> > > > > > >> >> >>> the
>> > > > > > >> >> >>>>>> email
>> > > > > > >> >> >>>>>>>> and any copies of it. Opinions, conclusion (etc)
>> > that
>> > > do
>> > > > > not
>> > > > > > >> >> relate
>> > > > > > >> >> >>>> to
>> > > > > > >> >> >>>>>> the
>> > > > > > >> >> >>>>>>>> official business of this company shall be
>> > understood
>> > > as
>> > > > > > >> neither
>> > > > > > >> >> >>>> given
>> > > > > > >> >> >>>>>> nor
>> > > > > > >> >> >>>>>>>> endorsed by it. IG is a trading name of IG Markets
>> > > > Limited
>> > > > > > (a
>> > > > > > >> >> >>> company
>> > > > > > >> >> >>>>>>>> registered in England and Wales, company number
>> > > > 04008957)
>> > > > > > and
>> > > > > > >> IG
>> > > > > > >> >> >>>> Index
>> > > > > > >> >> >>>>>>>> Limited (a company registered in England and
>> Wales,
>> > > > > company
>> > > > > > >> number
>> > > > > > >> >> >>>>>>>> 01190902). Registered address at Cannon Bridge
>> > House,
>> > > 25
>> > > > > > >> Dowgate
>> > > > > > >> >> >>>> Hill,
>> > > > > > >> >> >>>>>>>> London EC4R 2YA. Both IG Markets Limited (register
>> > > > number
>> > > > > > >> 195355)
>> > > > > > >> >> >>>> and IG
>> > > > > > >> >> >>>>>>>> Index Limited (register number 114059) are
>> > authorised
>> > > > and
>> > > > > > >> >> regulated
>> > > > > > >> >> >>>> by
>> > > > > > >> >> >>>>>> the
>> > > > > > >> >> >>>>>>>> Financial Conduct Authority.
>> > > > > > >> >> >>>>>>>>
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>>> --
>> > > > > > >> >> >>>>>> Gwen Shapira
>> > > > > > >> >> >>>>>> Product Manager | Confluent
>> > > > > > >> >> >>>>>> 650.450.2760 | @gwenshap
>> > > > > > >> >> >>>>>> Follow us: Twitter | blog
>> > > > > > >> >> >>>>>>
>> > > > > > >> >> >>>>
>> > > > > > >> >> >>>>
>> > > > > > >> >> >>>>
>> > > > > > >> >> >>>> --
>> > > > > > >> >> >>>> Gwen Shapira
>> > > > > > >> >> >>>> Product Manager | Confluent
>> > > > > > >> >> >>>> 650.450.2760 | @gwenshap
>> > > > > > >> >> >>>> Follow us: Twitter | blog
>> > > > > > >> >> >>>>
>> > > > > > >> >> >>>
>> > > > > > >> >> >>>
>> > > > > > >> >> >>>
>> > > > > > >> >> >>> --
>> > > > > > >> >> >>> Nacho (Ignacio) Solis
>> > > > > > >> >> >>> Kafka
>> > > > > > >> >> >>> nso...@linkedin.com
>> > > > > > >> >> >>>
>> > > > > > >> >> >
>> > > > > > >> >> >
>> > > > > > >> >> >
>> > > > > > >> >> > --
>> > > > > > >> >> > Gwen Shapira
>> > > > > > >> >> > Product Manager | Confluent
>> > > > > > >> >> > 650.450.2760 | @gwenshap
>> > > > > > >> >> > Follow us: Twitter | blog
>> > > > > > >> >>
>> > > > > > >> >>
>> > > > > > >>
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> --
>> > > > > > >> Gwen Shapira
>> > > > > > >> Product Manager | Confluent
>> > > > > > >> 650.450.2760 | @gwenshap
>> > > > > > >> Follow us: Twitter | blog
>> > > > > > >>
>> > > > > > >
>> > > > > > >
>> > > > > > The information contained in this email is strictly confidential
>> > and
>> > > > for
>> > > > > > the use of the addressee only, unless otherwise indicated. If you
>> > are
>> > > > not
>> > > > > > the intended recipient, please do not read, copy, use or disclose
>> > to
>> > > > > others
>> > > > > > this message or any attachment. Please also notify the sender by
>> > > > replying
>> > > > > > to this email or by telephone (+44(020 7896 0011) and then delete
>> > the
>> > > > > email
>> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
>> relate
>> > > to
>> > > > > the
>> > > > > > official business of this company shall be understood as neither
>> > > given
>> > > > > nor
>> > > > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> > company
>> > > > > > registered in England and Wales, company number 04008957) and IG
>> > > Index
>> > > > > > Limited (a company registered in England and Wales, company
>> number
>> > > > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> > > Hill,
>> > > > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> > and
>> > > > IG
>> > > > > > Index Limited (register number 114059) are authorised and
>> regulated
>> > > by
>> > > > > the
>> > > > > > Financial Conduct Authority.
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> The information contained in this email is strictly confidential and for
>> the use of the addressee only, unless otherwise indicated. If you are not
>> the intended recipient, please do not read, copy, use or disclose to others
>> this message or any attachment. Please also notify the sender by replying
>> to this email or by telephone (+44(020 7896 0011) and then delete the email
>> and any copies of it. Opinions, conclusion (etc) that do not relate to the
>> official business of this company shall be understood as neither given nor
>> endorsed by it. IG is a trading name of IG Markets Limited (a company
>> registered in England and Wales, company number 04008957) and IG Index
>> Limited (a company registered in England and Wales, company number
>> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> Index Limited (register number 114059) are authorised and regulated by the
>> Financial Conduct Authority.
>
> --
> Nacho - Ignacio Solis - iso...@igso.net
