Bumping up this thread. Is there a final verdict on this KIP?

On Sat, Dec 3, 2016 at 2:29 AM, Gwen Shapira <g...@confluent.io> wrote:

> Woah, I wasn't aware this is something we'll do. It wasn't in the KIP,
> right?
>
> I guess we could do it the same way ACLs currently work.
> I had in mind something that will allow admins to apply rules to the
> new create/delete/config topic APIs. So Todd can decide to reject
> "create topic" requests that ask for more than 40 partitions, or
> require exactly 3 replicas, or no more than 50GB partition size, etc.
>
> ACLs were added a bit ad-hoc, if we are planning to apply more rules
> to requests (and I think we should), we may want a bit more generic
> design around that.
>
> On Fri, Dec 2, 2016 at 7:16 AM, radai <radai.rosenbl...@gmail.com> wrote:
> > "wouldn't you be in the business of making sure everyone uses them
> > properly?"
> >
> > thats where a broker-side plugin would come handy - any incoming message
> > that does not conform to org policy (read - does not have the proper
> > headers) gets thrown out (with an error returned to user)
> >
> > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <tpal...@gmail.com> wrote:
> >
> >> Come on, I’ve done at least 2 talks on this one :)
> >>
> >> Producing counts to a topic is part of it, but that’s only part. So you
> >> count you have 100 messages in topic A. When you mirror topic A to
> another
> >> cluster, you have 99 messages. Where was your problem? Or worse, you
> have
> >> 100 messages, but one producer duplicated messages and another one lost
> >> messages. You need details about where the message came from in order to
> >> pinpoint problems when they happen. Source producer info, where it was
> >> produced into your infrastructure, and when it was produced. This
> requires
> >> you to add the information to the message.
> >>
> >> And yes, you still need to maintain your clients. So maybe my original
> >> example was not the best. My thoughts on not wanting to be responsible
> for
> >> message formats stands, because that’s very much separate from the
> client.
> >> As you know, we have our own internal client library that can insert the
> >> right headers, and right now inserts the right audit information into
> the
> >> message fields. If they exist, and assuming the message is Avro encoded.
> >> What if someone wants to use JSON instead for a good reason? What if
> user X
> >> wants to encrypt messages, but user Y does not? Maintaining the client
> >> library is still much easier than maintaining the message formats.
> >>
> >> -Todd
> >>
> >>
> >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <g...@confluent.io> wrote:
> >>
> >> > Based on your last sentence, consider me convinced :)
> >> >
> >> > I get why headers are critical for Mirroring (you need tags to prevent
> >> > loops and sometimes to route messages to the correct destination).
> >> > But why do you need headers to audit? We are auditing by producing
> >> > counts to a side topic (and I was under the impression you do the
> >> > same), so we never need to modify the message.
> >> >
> >> > Another thing - after we added headers, wouldn't you be in the
> >> > business of making sure everyone uses them properly? Making sure
> >> > everyone includes the right headers you need, not using the header
> >> > names you intend to use, etc. I don't think the "policing" business
> >> > will ever go away.
> >> >
> >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <tpal...@gmail.com>
> wrote:
> >> > > Got it. As an ops guy, I'm not very happy with the workaround. Avro
> >> means
> >> > > that I have to be concerned with the format of the messages in
> order to
> >> > run
> >> > > the infrastructure (audit, mirroring, etc.). That means that I have
> to
> >> > > handle the schemas, and I have to enforce rules about good formats.
> >> This
> >> > is
> >> > > not something I want to be in the business of, because I should be
> able
> >> > to
> >> > > run a service infrastructure without needing to be in the weeds of
> >> > dealing
> >> > > with customer data formats.
> >> > >
> >> > > Trust me, a sizable portion of my support time is spent dealing with
> >> > schema
> >> > > issues. I really would like to get away from that. Maybe I'd have
> more
> >> > time
> >> > > for other hobbies. Like writing. ;)
> >> > >
> >> > > -Todd
> >> > >
> >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <g...@confluent.io>
> wrote:
> >> > >
> >> > >> I'm pretty satisfied with the current workarounds (Avro container
> >> > >> format), so I'm not too excited about the extra work required to do
> >> > >> headers in Kafka. I absolutely don't mind it if you do it...
> >> > >> I think the Apache convention for "good idea, but not willing to
> put
> >> > >> any work toward it" is +0.5? anyway, that's what I was trying to
> >> > >> convey :)
> >> > >>
> >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino <tpal...@gmail.com>
> >> wrote:
> >> > >> > Well I guess my question for you, then, is what is holding you
> back
> >> > from
> >> > >> > full support for headers? What’s the bit that you’re missing that
> >> has
> >> > you
> >> > >> > under a full +1?
> >> > >> >
> >> > >> > -Todd
> >> > >> >
> >> > >> >
> >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen Shapira <g...@confluent.io>
> >> > wrote:
> >> > >> >
> >> > >> >> I know why people who support headers support them, and I've
> seen
> >> > what
> >> > >> >> the discussion is like.
> >> > >> >>
> >> > >> >> This is why I'm asking people who are against headers
> (especially
> >> > >> >> committers) what will make them change their mind - so we can
> get
> >> > this
> >> > >> >> part over one way or another.
> >> > >> >>
> >> > >> >> If I sound frustrated it is not at Radai, Jun or you (Todd)...
> I am
> >> > >> >> just looking for something concrete we can do to move the
> >> discussion
> >> > >> >> along to the yummy design details (which is the argument I
> really
> >> am
> >> > >> >> looking forward to).
> >> > >> >>
> >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd Palino <tpal...@gmail.com>
> >> > wrote:
> >> > >> >> > So, Gwen, to your question (even though I’m not a
> committer)...
> >> > >> >> >
> >> > >> >> > I have always been a strong supporter of introducing the
> concept
> >> > of an
> >> > >> >> > envelope to messages, which headers accomplishes. The message
> key
> >> > is
> >> > >> >> > already an example of a piece of envelope information. By
> >> > providing a
> >> > >> >> means
> >> > >> >> > to do this within Kafka itself, and not relying on use-case
> >> > specific
> >> > >> >> > implementations, you make it much easier for components to
> >> > >> interoperate.
> >> > >> >> It
> >> > >> >> > simplifies development of all these things (message routing,
> >> > auditing,
> >> > >> >> > encryption, etc.) because each one does not have to reinvent
> the
> >> > >> wheel.
> >> > >> >> >
> >> > >> >> > It also makes it much easier from a client point of view if
> the
> >> > >> headers
> >> > >> >> are
> >> > >> >> > defined as part of the protocol and/or message format in
> general
> >> > >> because
> >> > >> >> > you can easily produce and consume messages without having to
> >> take
> >> > >> into
> >> > >> >> > account specific cases. For example, I want to route messages,
> >> but
> >> > >> >> client A
> >> > >> >> > doesn’t support the way audit implemented headers, and client
> B
> >> > >> doesn’t
> >> > >> >> > support the way encryption or routing implemented headers, so
> now
> >> > my
> >> > >> >> > application has to create some really fragile (my autocorrect
> >> just
> >> > >> tried
> >> > >> >> to
> >> > >> >> > make that “tragic”, which is probably appropriate too) code to
> >> > strip
> >> > >> >> > everything off, rather than just consuming the messages,
> picking
> >> > out
> >> > >> the
> >> > >> >> 1
> >> > >> >> > or 2 headers it’s interested in, and performing its function.
> >> > >> >> >
> >> > >> >> > Honestly, this discussion has been going on for a long time,
> and
> >> > it’s
> >> > >> >> > always “Oh, you came up with 2 use cases, and yeah, those use
> >> cases
> >> > >> are
> >> > >> >> > real things that someone would want to do. Here’s an alternate
> >> way
> >> > to
> >> > >> >> > implement them so let’s not do headers.” If we have a few use
> >> cases
> >> > >> that
> >> > >> >> we
> >> > >> >> > actually came up with, you can be sure that over the next year
> >> > >> there’s a
> >> > >> >> > dozen others that we didn’t think of that someone would like
> to
> >> > do. I
> >> > >> >> > really think it’s time to stop rehashing this discussion and
> >> > instead
> >> > >> >> focus
> >> > >> >> > on a workable standard that we can adopt.
> >> > >> >> >
> >> > >> >> > -Todd
> >> > >> >> >
> >> > >> >> >
> >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd Palino <
> tpal...@gmail.com>
> >> > >> wrote:
> >> > >> >> >
> >> > >> >> >> C. per message encryption
> >> > >> >> >>> One drawback of this approach is that this significantly
> reduce
> >> > the
> >> > >> >> >>> effectiveness of compression, which happens on a set of
> >> > serialized
> >> > >> >> >>> messages. An alternative is to enable SSL for wire
> encryption
> >> and
> >> > >> rely
> >> > >> >> on
> >> > >> >> >>> the storage system (e.g. LUKS) for at rest encryption.
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >> Jun, this is not sufficient. While this does cover the case
> of
> >> > >> removing
> >> > >> >> a
> >> > >> >> >> drive from the system, it will not satisfy most compliance
> >> > >> requirements
> >> > >> >> for
> >> > >> >> >> encryption of data as whoever has access to the broker itself
> >> > still
> >> > >> has
> >> > >> >> >> access to the unencrypted data. For end-to-end encryption you
> >> > need to
> >> > >> >> >> encrypt at the producer, before it enters the system, and
> >> decrypt
> >> > at
> >> > >> the
> >> > >> >> >> consumer, after it exits the system.
> >> > >> >> >>
> >> > >> >> >> -Todd
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai <
> >> radai.rosenbl...@gmail.com
> >> > >
> >> > >> >> wrote:
> >> > >> >> >>
> >> > >> >> >>> another big plus of headers in the protocol is that it would
> >> > enable
> >> > >> >> rapid
> >> > >> >> >>> iteration on ideas outside of core kafka and would reduce
> the
> >> > >> number of
> >> > >> >> >>> future wire format changes required.
> >> > >> >> >>>
> >> > >> >> >>> a lot of what is currently a KIP represents use cases that
> are
> >> > not
> >> > >> 100%
> >> > >> >> >>> relevant to all users, and some of them require rather
> invasive
> >> > wire
> >> > >> >> >>> protocol changes. a thing a good recent example of this is
> >> > kip-98.
> >> > >> >> >>> tx-utilizing traffic is expected to be a very small
> fraction of
> >> > >> total
> >> > >> >> >>> traffic and yet the changes are invasive.
> >> > >> >> >>>
> >> > >> >> >>> every such wire format change translates into painful and
> slow
> >> > >> >> adoption of
> >> > >> >> >>> new versions.
> >> > >> >> >>>
> >> > >> >> >>> i think a lot of functionality currently in KIPs could be
> "spun
> >> > out"
> >> > >> >> and
> >> > >> >> >>> implemented as opt-in plugins transmitting data over
> headers.
> >> > this
> >> > >> >> would
> >> > >> >> >>> keep the core wire format stable(r), core codebase smaller,
> and
> >> > >> avoid
> >> > >> >> the
> >> > >> >> >>> "burden of proof" thats sometimes required to prove a
> certain
> >> > >> feature
> >> > >> >> is
> >> > >> >> >>> useful enough for a wide-enough audience to warrant a wire
> >> format
> >> > >> >> change
> >> > >> >> >>> and code complexity additions.
> >> > >> >> >>>
> >> > >> >> >>> (to be clear - kip-98 goes beyond "mere" wire format changes
> >> and
> >> > im
> >> > >> not
> >> > >> >> >>> saying it could have been completely done with headers, but
> >> > >> >> exactly-once
> >> > >> >> >>> delivery certainly could)
> >> > >> >> >>>
> >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen Shapira <
> >> g...@confluent.io
> >> > >
> >> > >> >> wrote:
> >> > >> >> >>>
> >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM, radai <
> >> > >> radai.rosenbl...@gmail.com>
> >> > >> >> >>> wrote:
> >> > >> >> >>> > > "For use cases within an organization, one could always
> use
> >> > >> other
> >> > >> >> >>> > > approaches such as company-wise containers"
> >> > >> >> >>> > > this is what linkedin has traditionally done but there
> are
> >> > now
> >> > >> >> cases
> >> > >> >> >>> > (read
> >> > >> >> >>> > > - topics) where this is not acceptable. this makes
> headers
> >> > >> useful
> >> > >> >> even
> >> > >> >> >>> > > within single orgs for cases where
> one-container-fits-all
> >> > cannot
> >> > >> >> >>> apply.
> >> > >> >> >>> > >
> >> > >> >> >>> > > as for the particular use cases listed, i dont want
> this to
> >> > >> devolve
> >> > >> >> >>> to a
> >> > >> >> >>> > > discussion of particular use cases - i think its enough
> >> that
> >> > >> some
> >> > >> >> of
> >> > >> >> >>> them
> >> > >> >> >>> >
> >> > >> >> >>> > I think a main point of contention is that: We identified
> few
> >> > >> >> >>> > use-cases where headers are useful, do we want Kafka to
> be a
> >> > >> system
> >> > >> >> >>> > that supports those use-cases?
> >> > >> >> >>> >
> >> > >> >> >>> > For example, Jun said:
> >> > >> >> >>> > "Not sure how widely useful record-level lineage is though
> >> > since
> >> > >> the
> >> > >> >> >>> > overhead could
> >> > >> >> >>> > be significant."
> >> > >> >> >>> >
> >> > >> >> >>> > We know NiFi supports record level lineage. I don't think
> it
> >> > was
> >> > >> >> >>> > developed for lols, I think it is safe to assume that the
> NSA
> >> > >> needed
> >> > >> >> >>> > that functionality. We also know that certain financial
> >> > institutes
> >> > >> >> >>> > need to track tampering with records at a record level and
> >> > there
> >> > >> are
> >> > >> >> >>> > federal regulations that absolutely require this.  They
> also
> >> > need
> >> > >> to
> >> > >> >> >>> > prove that routing apps that "touches" the messages and
> >> either
> >> > >> reads
> >> > >> >> >>> > or updates headers couldn't have possibly modified the
> >> payload
> >> > >> >> itself.
> >> > >> >> >>> > They use record level encryption to do that - apps can
> read
> >> and
> >> > >> >> >>> > (sometimes) modify headers but can't touch the payload.
> >> > >> >> >>> >
> >> > >> >> >>> > We can totally say "those are corner cases and not worth
> >> adding
> >> > >> >> >>> > headers to Kafka for", they should use a different pubsub
> >> > message
> >> > >> for
> >> > >> >> >>> > that (Nifi or one of the other 1000 that cater
> specifically
> >> to
> >> > the
> >> > >> >> >>> > financial industry).
> >> > >> >> >>> >
> >> > >> >> >>> > But this gets us into a catch 22:
> >> > >> >> >>> > If we discuss a specific use-case, someone can always say
> it
> >> > isn't
> >> > >> >> >>> > interesting enough for Kafka. If we discuss more general
> >> > trends,
> >> > >> >> >>> > others can say "well, we are not sure any of them really
> >> needs
> >> > >> >> headers
> >> > >> >> >>> > specifically. This is just hand waving and not
> interesting.".
> >> > >> >> >>> >
> >> > >> >> >>> > I think discussing use-cases in specifics is super
> important
> >> to
> >> > >> >> decide
> >> > >> >> >>> > implementation details for headers (my use-cases lean
> toward
> >> > >> >> numerical
> >> > >> >> >>> > keys with namespaces and object values, others differ),
> but I
> >> > >> think
> >> > >> >> we
> >> > >> >> >>> > need to answer the general "Are we going to have headers"
> >> > question
> >> > >> >> >>> > first.
> >> > >> >> >>> >
> >> > >> >> >>> > I'd love to hear from the other committers in the
> discussion:
> >> > >> >> >>> > What would it take to convince you that headers in Kafka
> are
> >> a
> >> > >> good
> >> > >> >> >>> > idea in general, so we can move ahead and try to agree on
> the
> >> > >> >> details?
> >> > >> >> >>> >
> >> > >> >> >>> > I feel like we keep moving the goal posts and this is
> truly
> >> > >> >> exhausting.
> >> > >> >> >>> >
> >> > >> >> >>> > For the record, I mildly support adding headers to Kafka
> >> > (+0.5?).
> >> > >> >> >>> > The community can continue to find workarounds to the
> issue
> >> and
> >> > >> there
> >> > >> >> >>> > are some benefits to keeping the message format and
> clients
> >> > >> simpler.
> >> > >> >> >>> > But I see the usefulness of headers to many use-cases and
> if
> >> we
> >> > >> can
> >> > >> >> >>> > find a good and generally useful way to add it to Kafka,
> it
> >> > will
> >> > >> make
> >> > >> >> >>> > Kafka easier to use for many - worthy goal in my eyes.
> >> > >> >> >>> >
> >> > >> >> >>> > > are interesting/feasible, but:
> >> > >> >> >>> > > A+B. i think there are use cases for polyglot topics.
> >> > >> especially if
> >> > >> >> >>> kafka
> >> > >> >> >>> > > is being used to "trunk" something else.
> >> > >> >> >>> > > D. multiple topics would make it harder to write
> portable
> >> > >> consumer
> >> > >> >> >>> code.
> >> > >> >> >>> > > partition remapping would mess with locality of
> consumption
> >> > >> >> >>> guarantees.
> >> > >> >> >>> > > E+F. a use case I see for lineage/metadata is
> >> > >> billing/chargeback.
> >> > >> >> for
> >> > >> >> >>> > that
> >> > >> >> >>> > > use case it is not enough to simply record the point of
> >> > origin,
> >> > >> but
> >> > >> >> >>> every
> >> > >> >> >>> > > replication stop (think mirror maker) must also add a
> >> record
> >> > to
> >> > >> >> form a
> >> > >> >> >>> > > "transit log".
> >> > >> >> >>> > >
> >> > >> >> >>> > > as for stream processing on top of kafka - i know samza
> >> has a
> >> > >> >> metadata
> >> > >> >> >>> > map
> >> > >> >> >>> > > which they carry around in addition to user values.
> headers
> >> > are
> >> > >> the
> >> > >> >> >>> > perfect
> >> > >> >> >>> > > fit for these things.
> >> > >> >> >>> > >
> >> > >> >> >>> > >
> >> > >> >> >>> > >
> >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM, Jun Rao <
> j...@confluent.io
> >> >
> >> > >> wrote:
> >> > >> >> >>> > >
> >> > >> >> >>> > >> Hi, Michael,
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> In order to answer the first two questions, it would be
> >> > helpful
> >> > >> >> if we
> >> > >> >> >>> > could
> >> > >> >> >>> > >> identify 1 or 2 strong use cases for headers in the
> space
> >> > for
> >> > >> >> >>> > third-party
> >> > >> >> >>> > >> vendors. For use cases within an organization, one
> could
> >> > always
> >> > >> >> use
> >> > >> >> >>> > other
> >> > >> >> >>> > >> approaches such as company-wise containers to get
> around
> >> w/o
> >> > >> >> >>> headers. I
> >> > >> >> >>> > >> went through the use cases in the KIP and in Radai's
> wiki
> >> (
> >> > >> >> >>> > >> https://cwiki.apache.org/confluence/display/KAFKA/A+
> >> > >> >> >>> > Case+for+Kafka+Headers
> >> > >> >> >>> > >> ).
> >> > >> >> >>> > >> The following are the ones that that I understand and
> >> could
> >> > be
> >> > >> in
> >> > >> >> the
> >> > >> >> >>> > >> third-party use case category.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> A. content-type
> >> > >> >> >>> > >> It seems that in general, content-type should be set at
> >> the
> >> > >> topic
> >> > >> >> >>> level.
> >> > >> >> >>> > >> Not sure if mixing messages with different content
> types
> >> > >> should be
> >> > >> >> >>> > >> encouraged.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> B. schema id
> >> > >> >> >>> > >> Since the value is mostly useless without schema id, it
> >> > seems
> >> > >> that
> >> > >> >> >>> > storing
> >> > >> >> >>> > >> the schema id together with serialized bytes in the
> value
> >> is
> >> > >> >> better?
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> C. per message encryption
> >> > >> >> >>> > >> One drawback of this approach is that this
> significantly
> >> > reduce
> >> > >> >> the
> >> > >> >> >>> > >> effectiveness of compression, which happens on a set of
> >> > >> serialized
> >> > >> >> >>> > >> messages. An alternative is to enable SSL for wire
> >> > encryption
> >> > >> and
> >> > >> >> >>> rely
> >> > >> >> >>> > on
> >> > >> >> >>> > >> the storage system (e.g. LUKS) for at rest encryption.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> D. cluster ID for mirroring across Kafka clusters
> >> > >> >> >>> > >> This is actually interesting. Today, to avoid
> introducing
> >> > >> cycles
> >> > >> >> when
> >> > >> >> >>> > doing
> >> > >> >> >>> > >> mirroring across data centers, one would either have to
> >> set
> >> > up
> >> > >> two
> >> > >> >> >>> Kafka
> >> > >> >> >>> > >> clusters (a local and an aggregate) per data center or
> >> > rename
> >> > >> >> topics.
> >> > >> >> >>> > >> Neither is ideal. With headers, the producer could tag
> >> each
> >> > >> >> message
> >> > >> >> >>> with
> >> > >> >> >>> > >> the producing cluster ID in the header. MirrorMaker
> could
> >> > then
> >> > >> >> avoid
> >> > >> >> >>> > >> mirroring messages to a cluster if they are tagged with
> >> the
> >> > >> same
> >> > >> >> >>> cluster
> >> > >> >> >>> > >> id.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> However, an alternative approach is to introduce sth
> like
> >> > >> >> >>> hierarchical
> >> > >> >> >>> > >> topic and store messages from different clusters in
> >> > different
> >> > >> >> >>> partitions
> >> > >> >> >>> > >> under the same topic. This approach avoids filtering
> out
> >> > >> unneeded
> >> > >> >> >>> data
> >> > >> >> >>> > and
> >> > >> >> >>> > >> makes offset preserving easier to support. It may make
> >> > >> compaction
> >> > >> >> >>> > trickier
> >> > >> >> >>> > >> though since the same key may show up in different
> >> > partitions.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> E. record-level lineage
> >> > >> >> >>> > >> For example, a source connector could store in the
> message
> >> > the
> >> > >> >> >>> metadata
> >> > >> >> >>> > >> (e.g. UUID) of the source record. Similarly, if a
> stream
> >> job
> >> > >> >> >>> transforms
> >> > >> >> >>> > >> messages from topic A to topic B, the library could
> >> include
> >> > the
> >> > >> >> >>> source
> >> > >> >> >>> > >> message offset in each of the transformed message in
> the
> >> > >> header.
> >> > >> >> Not
> >> > >> >> >>> > sure
> >> > >> >> >>> > >> how widely useful record-level lineage is though since
> the
> >> > >> >> overhead
> >> > >> >> >>> > could
> >> > >> >> >>> > >> be significant.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> F. auditing metadata
> >> > >> >> >>> > >> We could put things like clientId/host/user in the
> header
> >> in
> >> > >> each
> >> > >> >> >>> > message
> >> > >> >> >>> > >> for auditing. These metadata are really at the producer
> >> > level
> >> > >> >> though.
> >> > >> >> >>> > So, a
> >> > >> >> >>> > >> more efficient way is to only include a "producerId"
> per
> >> > >> message
> >> > >> >> and
> >> > >> >> >>> > send
> >> > >> >> >>> > >> the producerId -> metadata mapping independently.
> KIP-98
> >> is
> >> > >> >> actually
> >> > >> >> >>> > >> proposing including such a producerId natively in the
> >> > message.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> So, overall, I not sure that I am fully convinced of
> the
> >> > strong
> >> > >> >> >>> > third-party
> >> > >> >> >>> > >> use cases of headers yet. Perhaps we could discuss a
> bit
> >> > more
> >> > >> to
> >> > >> >> make
> >> > >> >> >>> > one
> >> > >> >> >>> > >> or two really convincing use cases.
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> Another orthogonal  question is whether header should
> be
> >> > >> exposed
> >> > >> >> in
> >> > >> >> >>> > stream
> >> > >> >> >>> > >> processing systems such Kafka stream, Samza, and Spark
> >> > >> streaming.
> >> > >> >> >>> > >> Currently, those systems just deal with key/value
> pairs.
> >> > >> Should we
> >> > >> >> >>> > expose a
> >> > >> >> >>> > >> third thing header there too or somehow map header to
> key
> >> or
> >> > >> >> value?
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> Thanks,
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> Jun
> >> > >> >> >>> > >>
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM, Michael Pearce <
> >> > >> >> >>> michael.pea...@ig.com>
> >> > >> >> >>> > >> wrote:
> >> > >> >> >>> > >>
> >> > >> >> >>> > >> > I assume, that after a period of a week, that there
> is
> >> no
> >> > >> >> concerns
> >> > >> >> >>> now
> >> > >> >> >>> > >> > with points 1, and 2 and now we have agreement that
> >> > headers
> >> > >> are
> >> > >> >> >>> useful
> >> > >> >> >>> > >> and
> >> > >> >> >>> > >> > needed in Kafka. As such if put to a KIP vote, this
> >> > wouldn’t
> >> > >> be
> >> > >> >> a
> >> > >> >> >>> > reason
> >> > >> >> >>> > >> to
> >> > >> >> >>> > >> > reject.
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> > @
> >> > >> >> >>> > >> > Ignacio on point 4).
> >> > >> >> >>> > >> > I think for purpose of getting this KIP moving past
> >> this,
> >> > we
> >> > >> can
> >> > >> >> >>> state
> >> > >> >> >>> > >> the
> >> > >> >> >>> > >> > key will be a 4 bytes space that can will be
> naturally
> >> > >> >> interpreted
> >> > >> >> >>> as
> >> > >> >> >>> > an
> >> > >> >> >>> > >> > Int32 (if namespacing is later wanted you can easily
> >> split
> >> > >> this
> >> > >> >> >>> into
> >> > >> >> >>> > two
> >> > >> >> >>> > >> > int16 spaces), from the wire protocol implementation
> >> this
> >> > >> makes
> >> > >> >> no
> >> > >> >> >>> > >> > difference I don’t believe. Is this reasonable to
> all?
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> > On 5) as per point 4 therefor happy we keep with 32
> >> bits.
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> > On 18/11/2016, 20:34, "ignacio.so...@gmail.com on
> >> behalf
> >> > of
> >> > >> >> >>> Ignacio
> >> > >> >> >>> > >> > Solis" <ignacio.so...@gmail.com on behalf of
> >> > iso...@igso.net
> >> > >> >
> >> > >> >> >>> wrote:
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     Summary:
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     3) Yes - Header value as byte[]
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     4a) Int,Int - No
> >> > >> >> >>> > >> >     4b) Int - Yes
> >> > >> >> >>> > >> >     4c) String - Reluctant maybe
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     5) I believe the header system should take a
> single
> >> > >> int.  I
> >> > >> >> >>> think
> >> > >> >> >>> > >> > 32bits is
> >> > >> >> >>> > >> >     a good size, if you want to interpret this as to
> >> 16bit
> >> > >> >> numbers
> >> > >> >> >>> in
> >> > >> >> >>> > the
> >> > >> >> >>> > >> > layer
> >> > >> >> >>> > >> >     above go right ahead.  If somebody wants to argue
> >> for
> >> > 16
> >> > >> >> bits
> >> > >> >> >>> or
> >> > >> >> >>> > 64
> >> > >> >> >>> > >> > bits of
> >> > >> >> >>> > >> >     header key space I would listen.
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     Discussion:
> >> > >> >> >>> > >> >     Dividing the key space into sub_key_1 and
> sub_key_2
> >> > >> makes no
> >> > >> >> >>> > sense to
> >> > >> >> >>> > >> > me at
> >> > >> >> >>> > >> >     this layer.  Are we going to start providing
> APIs to
> >> > get
> >> > >> all
> >> > >> >> >>> the
> >> > >> >> >>> > >> >     sub_key_1s? or all the sub_key_2s?  If there is
> no
> >> > >> >> >>> distinguishing
> >> > >> >> >>> > >> > functions
> >> > >> >> >>> > >> >     that are applied to each one then they should be
> a
> >> > single
> >> > >> >> >>> value.
> >> > >> >> >>> > At
> >> > >> >> >>> > >> > this
> >> > >> >> >>> > >> >     layer all we're doing is equality.
> >> > >> >> >>> > >> >     If the above layer wants to interpret this as 2,
> 3
> >> or
> >> > >> more
> >> > >> >> >>> values
> >> > >> >> >>> > >> > that's a
> >> > >> >> >>> > >> >     different question.  I personally think it's all
> one
> >> > >> >> keyspace
> >> > >> >> >>> > that is
> >> > >> >> >>> > >> >     getting assigned using some structure, but if you
> >> > want to
> >> > >> >> >>> > sub-assign
> >> > >> >> >>> > >> > parts
> >> > >> >> >>> > >> >     of it then that's fine.
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     The same discussion applies to strings.  If
> somebody
> >> > >> argued
> >> > >> >> for
> >> > >> >> >>> > >> > strings,
> >> > >> >> >>> > >> >     would we be arguing to divide the strings with
> dots
> >> > ('.')
> >> > >> >> as a
> >> > >> >> >>> > >> > requirement?
> >> > >> >> >>> > >> >     Would we want them to give us the different name
> >> > segments
> >> > >> >> >>> > separately?
> >> > >> >> >>> > >> >     Would we be performing any actions on this key
> other
> >> > than
> >> > >> >> >>> > matching?
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     Nacho
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     On Fri, Nov 18, 2016 at 9:30 AM, Michael Pearce <
> >> > >> >> >>> > >> michael.pea...@ig.com
> >> > >> >> >>> > >> > >
> >> > >> >> >>> > >> >     wrote:
> >> > >> >> >>> > >> >
> >> > >> >> >>> > >> >     > #jay #jun any concerns on 1 and 2 still?
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > @all
> >> > >> >> >>> > >> >     > To get this moving along a bit more I'd also
> like
> >> to
> >> > >> ask
> >> > >> >> to
> >> > >> >> >>> get
> >> > >> >> >>> > >> > clarity on
> >> > >> >> >>> > >> >     > the below last points:
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > 3) I believe we're all roughly happy with the
> >> header
> >> > >> value
> >> > >> >> >>> > being a
> >> > >> >> >>> > >> > byte[]?
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > 4) I believe consensus has been for an
> namespace
> >> > based
> >> > >> int
> >> > >> >> >>> > approach
> >> > >> >> >>> > >> >     > {int,int} for the key. Any objections if this
> is
> >> > what
> >> > >> we
> >> > >> >> go
> >> > >> >> >>> > with?
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > 5) as we have if assumption in (4)  is correct,
> >> > >> {int,int}
> >> > >> >> >>> keys.
> >> > >> >> >>> > >> >     > Should both int's be int16 or int32?
> >> > >> >> >>> > >> >     > I'm for them being int16(2 bytes) as combined
> is
> >> > space
> >> > >> of
> >> > >> >> >>> > 4bytes as
> >> > >> >> >>> > >> > per
> >> > >> >> >>> > >> >     > original and gives plenty of combinations for
> the
> >> > >> >> >>> foreseeable,
> >> > >> >> >>> > and
> >> > >> >> >>> > >> > keeps
> >> > >> >> >>> > >> >     > the overhead small.
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > Do we see any benefit in another kip call to
> >> discuss
> >> > >> >> these at
> >> > >> >> >>> > all?
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > Cheers
> >> > >> >> >>> > >> >     > Mike
> >> > >> >> >>> > >> >     > ________________________________________
> >> > >> >> >>> > >> >     > From: K Burstev <k.burs...@yandex.com>
> >> > >> >> >>> > >> >     > Sent: Friday, November 18, 2016 7:07:07 AM
> >> > >> >> >>> > >> >     > To: dev@kafka.apache.org
> >> > >> >> >>> > >> >     > Subject: Re: [DISCUSS] KIP-82 - Add Record
> Headers
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > For what it is worth also i agree. As a user:
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     >  1) Yes - Headers are worthwhile
> >> > >> >> >>> > >> >     >  2) Yes - Headers should be a top level option
> >> > >> >> >>> > >> >     >
> >> > >> >> >>> > >> >     > 14.11.2016, 21:15, "Ignacio Solis" <
> >> iso...@igso.net
> >> > >:
> >> > >> >> >>> > >> >     > > 1) Yes - Headers are worthwhile
> >> > >> >> >>> > >> >     > > 2) Yes - Headers should be a top level option
> >> > >> >> >>> > >> >     > >
> >> > >> >> >>> > >> >     > > On Mon, Nov 14, 2016 at 9:16 AM, Michael
> Pearce
> >> <
> >> > >> >> >>> > >> > michael.pea...@ig.com>
> >> > >> >> >>> > >> >     > > wrote:
> >> > >> >> >>> > >> >     > >
> >> > >> >> >>> > >> >     > >>  Hi Roger,
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  The kip details/examples the original
> proposal
> >> > for
> >> > >> key
> >> > >> >> >>> > spacing
> >> > >> >> >>> > >> ,
> >> > >> >> >>> > >> > not
> >> > >> >> >>> > >> >     > the
> >> > >> >> >>> > >> >     > >>  new mentioned as per discussion namespace
> >> idea.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  We will need to update the kip, when we get
> >> > >> agreement
> >> > >> >> >>> this
> >> > >> >> >>> > is a
> >> > >> >> >>> > >> > better
> >> > >> >> >>> > >> >     > >>  approach (which seems to be the case if I
> have
> >> > >> >> understood
> >> > >> >> >>> > the
> >> > >> >> >>> > >> > general
> >> > >> >> >>> > >> >     > >>  feeling in the conversation)
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Re the variable ints, at very early stage
> we
> >> did
> >> > >> think
> >> > >> >> >>> about
> >> > >> >> >>> > >> > this. I
> >> > >> >> >>> > >> >     > think
> >> > >> >> >>> > >> >     > >>  the added complexity for the saving isn't
> >> worth
> >> > it.
> >> > >> >> I'd
> >> > >> >> >>> > rather
> >> > >> >> >>> > >> go
> >> > >> >> >>> > >> >     > with, if
> >> > >> >> >>> > >> >     > >>  we want to reduce overheads and size int16
> >> > (2bytes)
> >> > >> >> keys
> >> > >> >> >>> as
> >> > >> >> >>> > it
> >> > >> >> >>> > >> > keeps it
> >> > >> >> >>> > >> >     > >>  simple.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  On the note of no headers, there is as per
> the
> >> > kip
> >> > >> as
> >> > >> >> we
> >> > >> >> >>> > use an
> >> > >> >> >>> > >> >     > attribute
> >> > >> >> >>> > >> >     > >>  bit to denote if headers are present or
> not as
> >> > such
> >> > >> >> >>> > provides a
> >> > >> >> >>> > >> > zero
> >> > >> >> >>> > >> >     > >>  overhead currently if headers are not used.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  I think as radai mentions would be good
> first
> >> > if we
> >> > >> >> can
> >> > >> >> >>> get
> >> > >> >> >>> > >> > clarity if
> >> > >> >> >>> > >> >     > do
> >> > >> >> >>> > >> >     > >>  we now have general consensus that (1)
> headers
> >> > are
> >> > >> >> >>> > worthwhile
> >> > >> >> >>> > >> and
> >> > >> >> >>> > >> >     > useful,
> >> > >> >> >>> > >> >     > >>  and (2) we want it as a top level entity.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Just to state the obvious i believe (1)
> >> headers
> >> > are
> >> > >> >> >>> > worthwhile
> >> > >> >> >>> > >> > and (2)
> >> > >> >> >>> > >> >     > >>  agree as a top level entity.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Cheers
> >> > >> >> >>> > >> >     > >>  Mike
> >> > >> >> >>> > >> >     > >>  ________________________________________
> >> > >> >> >>> > >> >     > >>  From: Roger Hoover <roger.hoo...@gmail.com
> >
> >> > >> >> >>> > >> >     > >>  Sent: Wednesday, November 9, 2016 9:10:47
> PM
> >> > >> >> >>> > >> >     > >>  To: dev@kafka.apache.org
> >> > >> >> >>> > >> >     > >>  Subject: Re: [DISCUSS] KIP-82 - Add Record
> >> > Headers
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Sorry for going a little in the weeds but
> >> thanks
> >> > >> for
> >> > >> >> the
> >> > >> >> >>> > >> replies
> >> > >> >> >>> > >> >     > regarding
> >> > >> >> >>> > >> >     > >>  varint.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Agreed that a prefix and {int, int} can be
> the
> >> > >> same.
> >> > >> >> It
> >> > >> >> >>> > doesn't
> >> > >> >> >>> > >> > look
> >> > >> >> >>> > >> >     > like
> >> > >> >> >>> > >> >     > >>  that's what the KIP is saying the "Open"
> >> > section.
> >> > >> The
> >> > >> >> >>> > example
> >> > >> >> >>> > >> > shows
> >> > >> >> >>> > >> >     > >>  2100001
> >> > >> >> >>> > >> >     > >>  for New Relic and 210002 for App Dynamics
> >> > implying
> >> > >> >> that
> >> > >> >> >>> the
> >> > >> >> >>> > New
> >> > >> >> >>> > >> > Relic
> >> > >> >> >>> > >> >     > >>  organization will have only a single
> header id
> >> > to
> >> > >> work
> >> > >> >> >>> > with. Or
> >> > >> >> >>> > >> > is
> >> > >> >> >>> > >> >     > 2100001
> >> > >> >> >>> > >> >     > >>  a prefix? The main point of a namespace or
> >> > prefix
> >> > >> is
> >> > >> >> to
> >> > >> >> >>> > reduce
> >> > >> >> >>> > >> > the
> >> > >> >> >>> > >> >     > >>  overhead of config mapping or registration
> >> > >> depending
> >> > >> >> on
> >> > >> >> >>> how
> >> > >> >> >>> > >> >     > >>  namespaces/prefixes are managed.
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Would love to hear more feedback on the
> >> > >> higher-level
> >> > >> >> >>> > questions
> >> > >> >> >>> > >> >     > though...
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Cheers,
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  Roger
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  On Wed, Nov 9, 2016 at 11:38 AM, radai <
> >> > >> >> >>> > >> > radai.rosenbl...@gmail.com>
> >> > >> >> >>> > >> >     > wrote:
> >> > >> >> >>> > >> >     > >>
> >> > >> >> >>> > >> >     > >>  > I think this discussion is getting a bit
> >> into
> >> > the
> >> > >> >> >>> weeds on
> >> > >> >> >>> > >> > technical
> >> > >> >> >>> > >> >     > >>  > implementation details.
> >> > >> >> >>> > >> >     > >>  > I'd liek to step back a minute and try
> and
> >> > >> establish
> >> > >> >> >>> > where we
> >> > >> >> >>> > >> > are in
> >> > >> >> >>> > >> >     > the
> >> > >> >> >>> > >> >     > >>  > larger picture:
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > (re-wording nacho's last paragraph)
> >> > >> >> >>> > >> >     > >>  > 1. are we all in agreement that headers
> are
> >> a
> >> > >> >> >>> worthwhile
> >> > >> >> >>> > and
> >> > >> >> >>> > >> > useful
> >> > >> >> >>> > >> >     > >>  > addition to have? this was contested
> early
> >> on
> >> > >> >> >>> > >> >     > >>  > 2. are we all in agreement on headers as
> top
> >> > >> level
> >> > >> >> >>> entity
> >> > >> >> >>> > vs
> >> > >> >> >>> > >> > headers
> >> > >> >> >>> > >> >     > >>  > squirreled-away in V?
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > if there are still concerns around these
> #2
> >> > >> points
> >> > >> >> >>> (#jay?
> >> > >> >> >>> > >> > #jun?)?
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > (and now back to our normal programming
> ...)
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > varints are nice. having said that, its
> >> adding
> >> > >> >> >>> complexity
> >> > >> >> >>> > >> (see
> >> > >> >> >>> > >> >     > >>  > https://github.com/addthis/
> >> > >> >> stream-lib/blob/master/src/
> >> > >> >> >>> > >> >     > >>  > main/java/com/clearspring/
> >> > >> >> analytics/util/Varint.java
> >> > >> >> >>> > >> >     > >>  > as 1st google result) and would require
> >> anyone
> >> > >> >> writing
> >> > >> >> >>> > other
> >> > >> >> >>> > >> > clients
> >> > >> >> >>> > >> >     > (C?
> >> > >> >> >>> > >> >     > >>  > Python? Go? Bash? ;-) ) to get/implement
> the
> >> > >> same,
> >> > >> >> and
> >> > >> >> >>> for
> >> > >> >> >>> > >> > relatively
> >> > >> >> >>> > >> >     > >>  > little gain (int vs string is order of
> >> > magnitude,
> >> > >> >> this
> >> > >> >> >>> > isnt).
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > int namespacing vs {int, int} namespacing
> >> are
> >> > >> >> basically
> >> > >> >> >>> > the
> >> > >> >> >>> > >> > same
> >> > >> >> >>> > >> >     > thing -
> >> > >> >> >>> > >> >     > >>  > youre just namespacing an int64 and
> giving
> >> > people
> >> > >> >> while
> >> > >> >> >>> > 2^32
> >> > >> >> >>> > >> > ranges
> >> > >> >> >>> > >> >     > at a
> >> > >> >> >>> > >> >     > >>  > time. the part i like about this is
> letting
> >> > >> people
> >> > >> >> >>> have a
> >> > >> >> >>> > >> large
> >> > >> >> >>> > >> >     > swath of
> >> > >> >> >>> > >> >     > >>  > numbers with one registration so they
> dont
> >> > have
> >> > >> to
> >> > >> >> come
> >> > >> >> >>> > back
> >> > >> >> >>> > >> > for
> >> > >> >> >>> > >> >     > every
> >> > >> >> >>> > >> >     > >>  > single plugin/header they want to
> "reserve".
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > On Wed, Nov 9, 2016 at 11:01 AM, Roger
> >> Hoover
> >> > <
> >> > >> >> >>> > >> >     > roger.hoo...@gmail.com>
> >> > >> >> >>> > >> >     > >>  > wrote:
> >> > >> >> >>> > >> >     > >>  >
> >> > >> >> >>> > >> >     > >>  > > Since some of the debate has been about
> >> > >> overhead +
> >> > >> >> >>> > >> > performance, I'm
> >> > >> >> >>> > >> >     > >>  > > wondering if we have considered a
> varint
> >> > >> encoding
> >> > >> >> (
> >> > >> >> >>> > >> >     > >>  > > https://developers.google.com/
> >> > >> >> protocol-buffers/docs/
> >> > >> >> >>> > >> >     > encoding#varints)
> >> > >> >> >>> > >> >     > >>  > for
> >> > >> >> >>> > >> >     > >>  > > the header length field (int32 in the
> >> > proposal)
> >> > >> >> and
> >> > >> >> >>> for
> >> > >> >> >>> > >> > header
> >> > >> >> >>> > >> >     > ids? If
> >> > >> >> >>> > >> >     > >>  > you
> >> > >> >> >>> > >> >     > >>  > > don't use headers, the overhead would
> be a
> >> > >> single
> >> > >> >> >>> byte
> >> > >> >> >>> > and
> >> > >> >> >>> > >> > for each
> >> > >> >> >>> > >> >     > >>  > header
> >> > >> >> >>> > >> >     > >>  > > id < 128 would also need only a single
> >> byte?
> >> > >> >> >>> > >> >     > >>  > >
> >> > >> >> >>> > >> >     > >>  > >
> >> > >> >> >>> > >> >     > >>  > >
> >> > >> >> >>> > >> >     > >>  > > On Wed, Nov 9, 2016 at 6:43 AM, radai <
> >> > >> >> >>> > >> > radai.rosenbl...@gmail.com>
> >> > >> >> >>> > >> >     > >>  > wrote:
> >> > >> >> >>> > >> >     > >>  > >
> >> > >> >> >>> > >> >     > >>  > > > @magnus - and very dangerous (youre
> >> > >> essentially
> >> > >> >> >>> > >> > downloading and
> >> > >> >> >>> > >> >     > >>  > executing
> >> > >> >> >>> > >> >     > >>  > > > arbitrary code off the internet on
> your
> >> > >> servers
> >> > >> >> ...
> >> > >> >> >>> > bad
> >> > >> >> >>> > >> > idea
> >> > >> >> >>> > >> >     > without
> >> > >> >> >>> > >> >     > >>  a
> >> > >> >> >>> > >> >     > >>  > > > sandbox, even with)
> >> > >> >> >>> > >> >     > >>  > > >
> >> > >> >> >>> > >> >     > >>  > > > as for it being a purely
> administrative
> >> > task
> >> > >> - i
> >> > >> >> >>> > >> disagree.
> >> > >> >> >>> > >> >     > >>  > > >
> >> > >> >> >>> > >> >     > >>  > > > i wish it would, really, because
> then my
> >> > >> earlier
> >> > >> >> >>> > point on
> >> > >> >> >>> > >> > the
> >> > >> >> >>> > >> >     > >>  > complexity
> >> > >> >> >>> > >> >     > >>  > > of
> >> > >> >> >>> > >> >     > >>  > > > the remapping process would be
> invalid,
> >> > but
> >> > >> at
> >> > >> >> >>> > linkedin,
> >> > >> >> >>> > >> > for
> >> > >> >> >>> > >> >     > example,
> >> > >> >> >>> > >> >     > >>  > we
> >> > >> >> >>> > >> >     > >>  > > > (the team im in) run kafka as a
> service.
> >> > we
> >> > >> dont
> >> > >> >> >>> > really
> >> > >> >> >>> > >> > know
> >> > >> >> >>> > >> >     > what our
> >> > >> >> >>> > >> >     > >>  > > users
> >> > >> >> >>> > >> >     > >>  > > > (developing applications that use
> kafka)
> >> > are
> >> > >> up
> >> > >> >> to
> >> > >> >> >>> at
> >> > >> >> >>> > any
> >> > >> >> >>> > >> > given
> >> > >> >> >>> > >> >     > >>  moment.
> >> > >> >> >>> > >> >     > >>  > > it
> >> > >> >> >>> > >> >     > >>  > > > is very possible (given the
> existance of
> >> > >> headers
> >> > >> >> >>> and a
> >> > >> >> >>> > >> >     > corresponding
> >> > >> >> >>> > >> >     > >>  > > plugin
> >> > >> >> >>> > >> >     > >>  > > > ecosystem) for some application to
> >> "equip"
> >> > >> their
> >> > >> >> >>> > >> producers
> >> > >> >> >>> > >> > and
> >> > >> >> >>> > >> >     > >>  > consumers
> >> > >> >> >>> > >> >     > >>  > > > with the required plugin without us
> >> > knowing.
> >> > >> i
> >> > >> >> dont
> >> > >> >> >>> > mean
> >> > >> >> >>> > >> > to imply
> >> > >> >> >>> > >> >     > >>  thats
> >> > >> >> >>> > >> >     > >>  > > > bad, i just want to make the point
> that
> >> > its
> >> > >> not
> >> > >> >> as
> >> > >> >> >>> > simple
> >> > >> >> >>> > >> >     > keeping it
> >> > >> >> >>> > >> >     > >>  in
> >> > >> >> >>> > >> >     > >>  > > > sync across a large-enough
> organization.
> >> > >> >> >>> > >> >     > >>  > > >
> >> > >> >> >>> > >> >     > >>  > > >
> >> > >> >> >>> > >> >     > >>  > > > On Wed, Nov 9, 2016 at 6:17 AM,
> Magnus
> >> > >> Edenhill
> >> > >> >> <
> >> > >> >> >>> > >> >     > mag...@edenhill.se>
> >> > >> >> >>> > >> >     > >>  > > > wrote:
> >> > >> >> >>> > >> >     > >>  > > >
> >> > >> >> >>> > >> >     > >>  > > > > I think there is a piece missing in
> >> the
> >> > >> >> Strings
> >> > >> >> >>> > >> > discussion,
> >> > >> >> >>> > >> >     > where
> >> > >> >> >>> > >> >     > >>  > > > > pro-Stringers
> >> > >> >> >>> > >> >     > >>  > > > > reason that by providing unique
> string
> >> > >> >> >>> identifiers
> >> > >> >> >>> > for
> >> > >> >> >>> > >> > each
> >> > >> >> >>> > >> >     > header
> >> > >> >> >>> > >> >     > >>  > > > > everything will just
> >> > >> >> >>> > >> >     > >>  > > > > magically work for all parts of the
> >> > stream
> >> > >> >> >>> pipeline.
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > But the strings dont mean anything
> by
> >> > >> >> themselves,
> >> > >> >> >>> > and
> >> > >> >> >>> > >> > while we
> >> > >> >> >>> > >> >     > >>  could
> >> > >> >> >>> > >> >     > >>  > > > > probably envision
> >> > >> >> >>> > >> >     > >>  > > > > some auto plugin loader that
> >> downloads,
> >> > >> >> compiles,
> >> > >> >> >>> > links
> >> > >> >> >>> > >> > and
> >> > >> >> >>> > >> >     > runs
> >> > >> >> >>> > >> >     > >>  > > plugins
> >> > >> >> >>> > >> >     > >>  > > > > on-demand
> >> > >> >> >>> > >> >     > >>  > > > > as soon as they're seen by a
> >> consumer, I
> >> > >> dont
> >> > >> >> >>> really
> >> > >> >> >>> > >> see
> >> > >> >> >>> > >> > a
> >> > >> >> >>> > >> >     > use-case
> >> > >> >> >>> > >> >     > >>  > for
> >> > >> >> >>> > >> >     > >>  > > > > something
> >> > >> >> >>> > >> >     > >>  > > > > so dynamic (and fragile) in
> practice.
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > In the real world an application
> will
> >> be
> >> > >> >> >>> configured
> >> > >> >> >>> > >> with
> >> > >> >> >>> > >> > a set
> >> > >> >> >>> > >> >     > of
> >> > >> >> >>> > >> >     > >>  > > plugins
> >> > >> >> >>> > >> >     > >>  > > > > to either add (producer)
> >> > >> >> >>> > >> >     > >>  > > > > or read (consumer) headers.
> >> > >> >> >>> > >> >     > >>  > > > > This is an administrative task
> based
> >> on
> >> > >> what
> >> > >> >> >>> > features a
> >> > >> >> >>> > >> > client
> >> > >> >> >>> > >> >     > >>  > > > > needs/provides and results in
> >> > >> >> >>> > >> >     > >>  > > > > some sort of configuration to
> enable
> >> and
> >> > >> >> >>> configure
> >> > >> >> >>> > the
> >> > >> >> >>> > >> > desired
> >> > >> >> >>> > >> >     > >>  > plugins.
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > Since this needs to be kept
> somewhat
> >> in
> >> > >> sync
> >> > >> >> >>> across
> >> > >> >> >>> > an
> >> > >> >> >>> > >> >     > organisation
> >> > >> >> >>> > >> >     > >>  > > > (there
> >> > >> >> >>> > >> >     > >>  > > > > is no point in having producers
> >> > >> >> >>> > >> >     > >>  > > > > add headers no consumers will read,
> >> and
> >> > >> vice
> >> > >> >> >>> versa),
> >> > >> >> >>> > >> the
> >> > >> >> >>> > >> > added
> >> > >> >> >>> > >> >     > >>  > > complexity
> >> > >> >> >>> > >> >     > >>  > > > > of assigning an id namespace
> >> > >> >> >>> > >> >     > >>  > > > > for each plugin as it is being
> >> > configured
> >> > >> >> should
> >> > >> >> >>> be
> >> > >> >> >>> > >> > tolerable.
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > /Magnus
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > 2016-11-09 13:06 GMT+01:00 Michael
> >> > Pearce <
> >> > >> >> >>> > >> >     > michael.pea...@ig.com>:
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > Just following/catching up on
> what
> >> > seems
> >> > >> to
> >> > >> >> be
> >> > >> >> >>> an
> >> > >> >> >>> > >> > active
> >> > >> >> >>> > >> >     > night :)
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > @Radai sorry if it may seem
> obvious
> >> > but
> >> > >> what
> >> > >> >> >>> does
> >> > >> >> >>> > MD
> >> > >> >> >>> > >> > stand
> >> > >> >> >>> > >> >     > for?
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > My take on String vs Int:
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > I will state first I am pro Int
> (16
> >> or
> >> > >> 32).
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > I do though playing devils
> advocate
> >> > see a
> >> > >> >> big
> >> > >> >> >>> plus
> >> > >> >> >>> > >> > with the
> >> > >> >> >>> > >> >     > >>  > argument
> >> > >> >> >>> > >> >     > >>  > > of
> >> > >> >> >>> > >> >     > >>  > > > > > String keys, this is around
> >> > integrating
> >> > >> >> into an
> >> > >> >> >>> > >> > existing
> >> > >> >> >>> > >> >     > >>  > eco-system.
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > As many other systems use String
> >> based
> >> > >> >> headers
> >> > >> >> >>> > >> (Flume,
> >> > >> >> >>> > >> > JMS)
> >> > >> >> >>> > >> >     > it
> >> > >> >> >>> > >> >     > >>  > makes
> >> > >> >> >>> > >> >     > >>  > > > it
> >> > >> >> >>> > >> >     > >>  > > > > > much easier for these to be
> >> > >> >> >>> > incorporated/integrated
> >> > >> >> >>> > >> > into.
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > How with Int based headers could
> we
> >> > >> provide
> >> > >> >> a
> >> > >> >> >>> > >> > way/guidence to
> >> > >> >> >>> > >> >     > >>  make
> >> > >> >> >>> > >> >     > >>  > > this
> >> > >> >> >>> > >> >     > >>  > > > > > integration simple / easy with
> >> > transition
> >> > >> >> flows
> >> > >> >> >>> > over
> >> > >> >> >>> > >> to
> >> > >> >> >>> > >> >     > kafka?
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > * tough luck buddy you're on your
> >> own
> >> > >> >> >>> > >> >     > >>  > > > > > * simply hash the string into int
> >> code
> >> > >> and
> >> > >> >> hope
> >> > >> >> >>> > for
> >> > >> >> >>> > >> no
> >> > >> >> >>> > >> >     > collisions
> >> > >> >> >>> > >> >     > >>  > > (how
> >> > >> >> >>> > >> >     > >>  > > > to
> >> > >> >> >>> > >> >     > >>  > > > > > convert back though?)
> >> > >> >> >>> > >> >     > >>  > > > > > * http2 style as mentioned by
> nacho.
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > cheers,
> >> > >> >> >>> > >> >     > >>  > > > > > Mike
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > ______________________________
> >> > __________
> >> > >> >> >>> > >> >     > >>  > > > > > From: radai <
> >> > radai.rosenbl...@gmail.com>
> >> > >> >> >>> > >> >     > >>  > > > > > Sent: Wednesday, November 9, 2016
> >> > 8:12 AM
> >> > >> >> >>> > >> >     > >>  > > > > > To: dev@kafka.apache.org
> >> > >> >> >>> > >> >     > >>  > > > > > Subject: Re: [DISCUSS] KIP-82 -
> Add
> >> > >> Record
> >> > >> >> >>> Headers
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > thinking about it some more, the
> >> best
> >> > >> way to
> >> > >> >> >>> > transmit
> >> > >> >> >>> > >> > the
> >> > >> >> >>> > >> >     > header
> >> > >> >> >>> > >> >     > >>  > > > > remapping
> >> > >> >> >>> > >> >     > >>  > > > > > data to consumers would be to
> put it
> >> > in
> >> > >> the
> >> > >> >> MD
> >> > >> >> >>> > >> response
> >> > >> >> >>> > >> >     > payload,
> >> > >> >> >>> > >> >     > >>  so
> >> > >> >> >>> > >> >     > >>  > > > maybe
> >> > >> >> >>> > >> >     > >>  > > > > > it should be discussed now.
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > On Wed, Nov 9, 2016 at 12:09 AM,
> >> > radai <
> >> > >> >> >>> > >> >     > >>  radai.rosenbl...@gmail.com
> >> > >> >> >>> > >> >     > >>  > >
> >> > >> >> >>> > >> >     > >>  > > > > wrote:
> >> > >> >> >>> > >> >     > >>  > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > > im not opposed to the idea of
> >> > namespace
> >> > >> >> >>> mapping.
> >> > >> >> >>> > >> all
> >> > >> >> >>> > >> > im
> >> > >> >> >>> > >> >     > saying
> >> > >> >> >>> > >> >     > >>  is
> >> > >> >> >>> > >> >     > >>  > > > that
> >> > >> >> >>> > >> >     > >>  > > > > > its
> >> > >> >> >>> > >> >     > >>  > > > > > > not part of the "mvp" and,
> since
> >> it
> >> > >> >> requires
> >> > >> >> >>> no
> >> > >> >> >>> > >> wire
> >> > >> >> >>> > >> > format
> >> > >> >> >>> > >> >     > >>  > change,
> >> > >> >> >>> > >> >     > >>  > > > can
> >> > >> >> >>> > >> >     > >>  > > > > > > always be added later.
> >> > >> >> >>> > >> >     > >>  > > > > > > also, its not as simple as just
> >> > >> >> configuring
> >> > >> >> >>> MM
> >> > >> >> >>> > to
> >> > >> >> >>> > >> do
> >> > >> >> >>> > >> > the
> >> > >> >> >>> > >> >     > >>  > transform:
> >> > >> >> >>> > >> >     > >>  > > > > lets
> >> > >> >> >>> > >> >     > >>  > > > > > > say i've implemented large
> message
> >> > >> >> support as
> >> > >> >> >>> > >> > {666,1} and
> >> > >> >> >>> > >> >     > on
> >> > >> >> >>> > >> >     > >>  some
> >> > >> >> >>> > >> >     > >>  > > > > mirror
> >> > >> >> >>> > >> >     > >>  > > > > > > target cluster its been
> remapped
> >> to
> >> > >> >> {999,1}.
> >> > >> >> >>> the
> >> > >> >> >>> > >> > consumer
> >> > >> >> >>> > >> >     > >>  plugin
> >> > >> >> >>> > >> >     > >>  > > code
> >> > >> >> >>> > >> >     > >>  > > > > > would
> >> > >> >> >>> > >> >     > >>  > > > > > > also need to be told to look
> for
> >> the
> >> > >> large
> >> > >> >> >>> > message
> >> > >> >> >>> > >> > "part X
> >> > >> >> >>> > >> >     > of
> >> > >> >> >>> > >> >     > >>  Y"
> >> > >> >> >>> > >> >     > >>  > > > header
> >> > >> >> >>> > >> >     > >>  > > > > > > under {999,1}. doable, but
> tricky.
> >> > >> >> >>> > >> >     > >>  > > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > > On Tue, Nov 8, 2016 at 10:29
> PM,
> >> > Gwen
> >> > >> >> >>> Shapira <
> >> > >> >> >>> > >> >     > >>  g...@confluent.io
> >> > >> >> >>> > >> >     > >>  > >
> >> > >> >> >>> > >> >     > >>  > > > > wrote:
> >> > >> >> >>> > >> >     > >>  > > > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > >> While you can do whatever you
> >> want
> >> > >> with a
> >> > >> >> >>> > >> namespace
> >> > >> >> >>> > >> > and
> >> > >> >> >>> > >> >     > your
> >> > >> >> >>> > >> >     > >>  > code,
> >> > >> >> >>> > >> >     > >>  > > > > > >> what I'd expect is for each
> app
> >> to
> >> > >> >> >>> namespaces
> >> > >> >> >>> > >> >     > configurable...
> >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> So if I accidentally used 666
> for
> >> > my
> >> > >> HR
> >> > >> >> >>> > >> department,
> >> > >> >> >>> > >> > and
> >> > >> >> >>> > >> >     > still
> >> > >> >> >>> > >> >     > >>  > want
> >> > >> >> >>> > >> >     > >>  > > > to
> >> > >> >> >>> > >> >     > >>  > > > > > >> run RadaiApp, I can config
> >> > >> "namespace=42"
> >> > >> >> >>> for
> >> > >> >> >>> > >> > RadaiApp and
> >> > >> >> >>> > >> >     > >>  > > > everything
> >> > >> >> >>> > >> >     > >>  > > > > > >> will look normal.
> >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> This means you only need to
> sync
> >> > usage
> >> > >> >> >>> inside
> >> > >> >> >>> > your
> >> > >> >> >>> > >> > own
> >> > >> >> >>> > >> >     > >>  > > organization.
> >> > >> >> >>> > >> >     > >>  > > > > > >> Still hard, but somewhat
> easier
> >> > than
> >> > >> >> syncing
> >> > >> >> >>> > with
> >> > >> >> >>> > >> > the
> >> > >> >> >>> > >> >     > entire
> >> > >> >> >>> > >> >     > >>  > > world.
> >> > >> >> >>> > >> >     > >>  > > > > > >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> On Tue, Nov 8, 2016 at 10:07
> PM,
> >> > >> radai <
> >> > >> >> >>> > >> >     > >>  > > radai.rosenbl...@gmail.com>
> >> > >> >> >>> > >> >     > >>  > > > > > >> wrote:
> >> > >> >> >>> > >> >     > >>  > > > > > >> > and we can start with
> >> {namespace,
> >> > >> id}
> >> > >> >> and
> >> > >> >> >>> no
> >> > >> >> >>> > >> > re-mapping
> >> > >> >> >>> > >> >     > >>  > support
> >> > >> >> >>> > >> >     > >>  > > > and
> >> > >> >> >>> > >> >     > >>  > > > > > >> always
> >> > >> >> >>> > >> >     > >>  > > > > > >> > add it later on if/when
> >> > collisions
> >> > >> >> >>> actually
> >> > >> >> >>> > >> > happen (i
> >> > >> >> >>> > >> >     > dont
> >> > >> >> >>> > >> >     > >>  > think
> >> > >> >> >>> > >> >     > >>  > > > > > they'd
> >> > >> >> >>> > >> >     > >>  > > > > > >> be
> >> > >> >> >>> > >> >     > >>  > > > > > >> > a problem).
> >> > >> >> >>> > >> >     > >>  > > > > > >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> > every interested party (so
> orgs
> >> > or
> >> > >> >> >>> > individuals)
> >> > >> >> >>> > >> > could
> >> > >> >> >>> > >> >     > then
> >> > >> >> >>> > >> >     > >>  > > > register
> >> > >> >> >>> > >> >     > >>  > > > > a
> >> > >> >> >>> > >> >     > >>  > > > > > >> > prefix (0 = reserved, 1 =
> >> > confluent
> >> > >> ...
> >> > >> >> >>> 666
> >> > >> >> >>> > = me
> >> > >> >> >>> > >> > :-) )
> >> > >> >> >>> > >> >     > and
> >> > >> >> >>> > >> >     > >>  do
> >> > >> >> >>> > >> >     > >>  > > > > whatever
> >> > >> >> >>> > >> >     > >>  > > > > > >> with
> >> > >> >> >>> > >> >     > >>  > > > > > >> > the 2nd ID - so once
> linkedin
> >> > >> >> registers,
> >> > >> >> >>> say
> >> > >> >> >>> > 3,
> >> > >> >> >>> > >> > then
> >> > >> >> >>> > >> >     > >>  linkedin
> >> > >> >> >>> > >> >     > >>  > > devs
> >> > >> >> >>> > >> >     > >>  > > > > are
> >> > >> >> >>> > >> >     > >>  > > > > > >> free
> >> > >> >> >>> > >> >     > >>  > > > > > >> > to use {3, *} with a
> reasonable
> >> > >> >> >>> expectation
> >> > >> >> >>> > to
> >> > >> >> >>> > >> to
> >> > >> >> >>> > >> >     > collide
> >> > >> >> >>> > >> >     > >>  with
> >> > >> >> >>> > >> >     > >>  > > > > > anything
> >> > >> >> >>> > >> >     > >>  > > > > > >> > else. further partitioning
> of
> >> > that *
> >> > >> >> >>> becomes
> >> > >> >> >>> > >> > linkedin's
> >> > >> >> >>> > >> >     > >>  > problem,
> >> > >> >> >>> > >> >     > >>  > > > but
> >> > >> >> >>> > >> >     > >>  > > > > > the
> >> > >> >> >>> > >> >     > >>  > > > > > >> > "upstream registration" of a
> >> > >> namespace
> >> > >> >> >>> only
> >> > >> >> >>> > has
> >> > >> >> >>> > >> to
> >> > >> >> >>> > >> >     > happen
> >> > >> >> >>> > >> >     > >>  > once.
> >> > >> >> >>> > >> >     > >>  > > > > > >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> > On Tue, Nov 8, 2016 at 9:03
> PM,
> >> > >> James
> >> > >> >> >>> Cheng <
> >> > >> >> >>> > >> >     > >>  > > wushuja...@gmail.com
> >> > >> >> >>> > >> >     > >>  > > > >
> >> > >> >> >>> > >> >     > >>  > > > > > >> wrote:
> >> > >> >> >>> > >> >     > >>  > > > > > >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > On Nov 8, 2016, at 5:54
> PM,
> >> > Gwen
> >> > >> >> >>> Shapira <
> >> > >> >> >>> > >> >     > >>  > g...@confluent.io>
> >> > >> >> >>> > >> >     > >>  > > > > > wrote:
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > Thank you so much for
> this
> >> > clear
> >> > >> and
> >> > >> >> >>> fair
> >> > >> >> >>> > >> > summary of
> >> > >> >> >>> > >> >     > the
> >> > >> >> >>> > >> >     > >>  > > > > arguments.
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > I'm in favor of ints.
> Not a
> >> > >> >> >>> deal-breaker,
> >> > >> >> >>> > but
> >> > >> >> >>> > >> > in
> >> > >> >> >>> > >> >     > favor.
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > Even more in favor of
> >> Magnus's
> >> > >> >> >>> > decentralized
> >> > >> >> >>> > >> >     > suggestion
> >> > >> >> >>> > >> >     > >>  > with
> >> > >> >> >>> > >> >     > >>  > > > > > Roger's
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > tweak: add a namespace
> for
> >> > >> headers.
> >> > >> >> >>> This
> >> > >> >> >>> > will
> >> > >> >> >>> > >> > allow
> >> > >> >> >>> > >> >     > each
> >> > >> >> >>> > >> >     > >>  > app
> >> > >> >> >>> > >> >     > >>  > > to
> >> > >> >> >>> > >> >     > >>  > > > > > just
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > use whatever IDs it wants
> >> > >> >> internally,
> >> > >> >> >>> and
> >> > >> >> >>> > >> then
> >> > >> >> >>> > >> > let
> >> > >> >> >>> > >> >     > the
> >> > >> >> >>> > >> >     > >>  > admin
> >> > >> >> >>> > >> >     > >>  > > > > > >> deploying
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > the app figure out an
> >> > available
> >> > >> >> >>> namespace
> >> > >> >> >>> > ID
> >> > >> >> >>> > >> > for the
> >> > >> >> >>> > >> >     > app
> >> > >> >> >>> > >> >     > >>  to
> >> > >> >> >>> > >> >     > >>  > > > live
> >> > >> >> >>> > >> >     > >>  > > > > > in.
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > So
> >> > io.confluent.schema-registry
> >> > >> can
> >> > >> >> be
> >> > >> >> >>> > >> > namespace
> >> > >> >> >>> > >> >     > 0x01 on
> >> > >> >> >>> > >> >     > >>  my
> >> > >> >> >>> > >> >     > >>  > > > > > >> deployment
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > and 0x57 on yours, and
> the
> >> > poor
> >> > >> guys
> >> > >> >> >>> > >> > developing the
> >> > >> >> >>> > >> >     > app
> >> > >> >> >>> > >> >     > >>  > don't
> >> > >> >> >>> > >> >     > >>  > > > > need
> >> > >> >> >>> > >> >     > >>  > > > > > to
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > worry about that.
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> Gwen, if I understand your
> >> > example
> >> > >> >> >>> right, an
> >> > >> >> >>> > >> >     > application
> >> > >> >> >>> > >> >     > >>  > > deployer
> >> > >> >> >>> > >> >     > >>  > > > > > might
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> decide to use 0x01 in one
> >> > >> deployment,
> >> > >> >> and
> >> > >> >> >>> > that
> >> > >> >> >>> > >> > means
> >> > >> >> >>> > >> >     > that
> >> > >> >> >>> > >> >     > >>  > once
> >> > >> >> >>> > >> >     > >>  > > > the
> >> > >> >> >>> > >> >     > >>  > > > > > >> message
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> is written into the
> broker, it
> >> > >> will be
> >> > >> >> >>> > saved on
> >> > >> >> >>> > >> > the
> >> > >> >> >>> > >> >     > broker
> >> > >> >> >>> > >> >     > >>  > with
> >> > >> >> >>> > >> >     > >>  > > > > that
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> specific namespace (0x01).
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> If you were to mirror that
> >> > message
> >> > >> >> into
> >> > >> >> >>> > another
> >> > >> >> >>> > >> >     > cluster,
> >> > >> >> >>> > >> >     > >>  the
> >> > >> >> >>> > >> >     > >>  > > 0x01
> >> > >> >> >>> > >> >     > >>  > > > > > would
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> accompany the message,
> right?
> >> > What
> >> > >> if
> >> > >> >> the
> >> > >> >> >>> > >> > deployers of
> >> > >> >> >>> > >> >     > the
> >> > >> >> >>> > >> >     > >>  > same
> >> > >> >> >>> > >> >     > >>  > > > app
> >> > >> >> >>> > >> >     > >>  > > > > > in
> >> > >> >> >>> > >> >     > >>  > > > > > >> the
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> other cluster uses 0x57?
> They
> >> > won't
> >> > >> >> >>> > understand
> >> > >> >> >>> > >> > each
> >> > >> >> >>> > >> >     > other?
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> I'm not sure that's an
> >> avoidable
> >> > >> >> >>> problem. I
> >> > >> >> >>> > >> > think it
> >> > >> >> >>> > >> >     > simply
> >> > >> >> >>> > >> >     > >>  > > means
> >> > >> >> >>> > >> >     > >>  > > > > > that
> >> > >> >> >>> > >> >     > >>  > > > > > >> in
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> order to share data, you
> have
> >> to
> >> > >> also
> >> > >> >> >>> have a
> >> > >> >> >>> > >> > shared
> >> > >> >> >>> > >> >     > (agreed
> >> > >> >> >>> > >> >     > >>  > > upon)
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> understanding of what the
> >> > >> namespaces
> >> > >> >> >>> mean.
> >> > >> >> >>> > >> Which
> >> > >> >> >>> > >> > I
> >> > >> >> >>> > >> >     > think
> >> > >> >> >>> > >> >     > >>  > makes
> >> > >> >> >>> > >> >     > >>  > > > > sense,
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> because the alternate
> (sharing
> >> > >> >> *nothing*
> >> > >> >> >>> at
> >> > >> >> >>> > >> all)
> >> > >> >> >>> > >> > would
> >> > >> >> >>> > >> >     > mean
> >> > >> >> >>> > >> >     > >>  > > that
> >> > >> >> >>> > >> >     > >>  > > > > > there
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> would be no way to
> understand
> >> > each
> >> > >> >> other.
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> -James
> >> > >> >> >>> > >> >     > >>  > > > > > >> >>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > Gwen
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> > On Tue, Nov 8, 2016 at
> 4:23
> >> > PM,
> >> > >> >> radai <
> >> > >> >> >>> > >> >     > >>  > > > > radai.rosenbl...@gmail.com>
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> wrote:
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >> +1 for sean's document.
> it
> >> > >> covers
> >> > >> >> >>> pretty
> >> > >> >> >>> > >> much
> >> > >> >> >>> > >> > all
> >> > >> >> >>> > >> >     > the
> >> > >> >> >>> > >> >     > >>  > > > trade-offs
> >> > >> >> >>> > >> >     > >>  > > > > > and
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >> provides concrete
> figures
> >> to
> >> > >> argue
> >> > >> >> >>> about
> >> > >> >> >>> > :-)
> >> > >> >> >>> > >> >     > >>  > > > > > >> >> >> (nit-picking - used the
> >> same
> >> > >> xkcd
> >> > >> >> >>> twice,
> >> > >> >> >>> > >> also
> >> > >> >> >>> > >> > trove
> >> > >> >> >>> > >> >     > has
> >> > >> >> >>> > >> >     > >>  > been
> >> > >> >> >>> > >> >     > >>  > > > > > >> superceded
> >> > >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Gwen Shapira
> >> > Product Manager | Confluent
> >> > 650.450.2760 | @gwenshap
> >> > Follow us: Twitter | blog
> >> >
> >>
> >>
> >>
> >> --
> >> *Todd Palino*
> >> Staff Site Reliability Engineer
> >> Data Infrastructure Streaming
> >>
> >>
> >>
> >> linkedin.com/in/toddpalino
> >>
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>



-- 
Regards,
Abhishek Agarwal

Reply via email to