Woah, I wasn't aware this is something we'll do. It wasn't in the KIP, right?
I guess we could do it the same way ACLs currently work. I had in mind something that will allow admins to apply rules to the new create/delete/config topic APIs. So Todd can decide to reject "create topic" requests that ask for more than 40 partitions, or require exactly 3 replicas, or no more than 50GB partition size, etc. ACLs were added a bit ad-hoc, if we are planning to apply more rules to requests (and I think we should), we may want a bit more generic design around that. On Fri, Dec 2, 2016 at 7:16 AM, radai <radai.rosenbl...@gmail.com> wrote: > "wouldn't you be in the business of making sure everyone uses them > properly?" > > thats where a broker-side plugin would come handy - any incoming message > that does not conform to org policy (read - does not have the proper > headers) gets thrown out (with an error returned to user) > > On Thu, Dec 1, 2016 at 8:44 PM, Todd Palino <tpal...@gmail.com> wrote: > >> Come on, I’ve done at least 2 talks on this one :) >> >> Producing counts to a topic is part of it, but that’s only part. So you >> count you have 100 messages in topic A. When you mirror topic A to another >> cluster, you have 99 messages. Where was your problem? Or worse, you have >> 100 messages, but one producer duplicated messages and another one lost >> messages. You need details about where the message came from in order to >> pinpoint problems when they happen. Source producer info, where it was >> produced into your infrastructure, and when it was produced. This requires >> you to add the information to the message. >> >> And yes, you still need to maintain your clients. So maybe my original >> example was not the best. My thoughts on not wanting to be responsible for >> message formats stands, because that’s very much separate from the client. >> As you know, we have our own internal client library that can insert the >> right headers, and right now inserts the right audit information into the >> message fields. If they exist, and assuming the message is Avro encoded. >> What if someone wants to use JSON instead for a good reason? What if user X >> wants to encrypt messages, but user Y does not? Maintaining the client >> library is still much easier than maintaining the message formats. >> >> -Todd >> >> >> On Thu, Dec 1, 2016 at 6:21 PM, Gwen Shapira <g...@confluent.io> wrote: >> >> > Based on your last sentence, consider me convinced :) >> > >> > I get why headers are critical for Mirroring (you need tags to prevent >> > loops and sometimes to route messages to the correct destination). >> > But why do you need headers to audit? We are auditing by producing >> > counts to a side topic (and I was under the impression you do the >> > same), so we never need to modify the message. >> > >> > Another thing - after we added headers, wouldn't you be in the >> > business of making sure everyone uses them properly? Making sure >> > everyone includes the right headers you need, not using the header >> > names you intend to use, etc. I don't think the "policing" business >> > will ever go away. >> > >> > On Thu, Dec 1, 2016 at 5:25 PM, Todd Palino <tpal...@gmail.com> wrote: >> > > Got it. As an ops guy, I'm not very happy with the workaround. Avro >> means >> > > that I have to be concerned with the format of the messages in order to >> > run >> > > the infrastructure (audit, mirroring, etc.). That means that I have to >> > > handle the schemas, and I have to enforce rules about good formats. >> This >> > is >> > > not something I want to be in the business of, because I should be able >> > to >> > > run a service infrastructure without needing to be in the weeds of >> > dealing >> > > with customer data formats. >> > > >> > > Trust me, a sizable portion of my support time is spent dealing with >> > schema >> > > issues. I really would like to get away from that. Maybe I'd have more >> > time >> > > for other hobbies. Like writing. ;) >> > > >> > > -Todd >> > > >> > > On Thu, Dec 1, 2016 at 4:04 PM Gwen Shapira <g...@confluent.io> wrote: >> > > >> > >> I'm pretty satisfied with the current workarounds (Avro container >> > >> format), so I'm not too excited about the extra work required to do >> > >> headers in Kafka. I absolutely don't mind it if you do it... >> > >> I think the Apache convention for "good idea, but not willing to put >> > >> any work toward it" is +0.5? anyway, that's what I was trying to >> > >> convey :) >> > >> >> > >> On Thu, Dec 1, 2016 at 3:05 PM, Todd Palino <tpal...@gmail.com> >> wrote: >> > >> > Well I guess my question for you, then, is what is holding you back >> > from >> > >> > full support for headers? What’s the bit that you’re missing that >> has >> > you >> > >> > under a full +1? >> > >> > >> > >> > -Todd >> > >> > >> > >> > >> > >> > On Thu, Dec 1, 2016 at 1:59 PM, Gwen Shapira <g...@confluent.io> >> > wrote: >> > >> > >> > >> >> I know why people who support headers support them, and I've seen >> > what >> > >> >> the discussion is like. >> > >> >> >> > >> >> This is why I'm asking people who are against headers (especially >> > >> >> committers) what will make them change their mind - so we can get >> > this >> > >> >> part over one way or another. >> > >> >> >> > >> >> If I sound frustrated it is not at Radai, Jun or you (Todd)... I am >> > >> >> just looking for something concrete we can do to move the >> discussion >> > >> >> along to the yummy design details (which is the argument I really >> am >> > >> >> looking forward to). >> > >> >> >> > >> >> On Thu, Dec 1, 2016 at 1:53 PM, Todd Palino <tpal...@gmail.com> >> > wrote: >> > >> >> > So, Gwen, to your question (even though I’m not a committer)... >> > >> >> > >> > >> >> > I have always been a strong supporter of introducing the concept >> > of an >> > >> >> > envelope to messages, which headers accomplishes. The message key >> > is >> > >> >> > already an example of a piece of envelope information. By >> > providing a >> > >> >> means >> > >> >> > to do this within Kafka itself, and not relying on use-case >> > specific >> > >> >> > implementations, you make it much easier for components to >> > >> interoperate. >> > >> >> It >> > >> >> > simplifies development of all these things (message routing, >> > auditing, >> > >> >> > encryption, etc.) because each one does not have to reinvent the >> > >> wheel. >> > >> >> > >> > >> >> > It also makes it much easier from a client point of view if the >> > >> headers >> > >> >> are >> > >> >> > defined as part of the protocol and/or message format in general >> > >> because >> > >> >> > you can easily produce and consume messages without having to >> take >> > >> into >> > >> >> > account specific cases. For example, I want to route messages, >> but >> > >> >> client A >> > >> >> > doesn’t support the way audit implemented headers, and client B >> > >> doesn’t >> > >> >> > support the way encryption or routing implemented headers, so now >> > my >> > >> >> > application has to create some really fragile (my autocorrect >> just >> > >> tried >> > >> >> to >> > >> >> > make that “tragic”, which is probably appropriate too) code to >> > strip >> > >> >> > everything off, rather than just consuming the messages, picking >> > out >> > >> the >> > >> >> 1 >> > >> >> > or 2 headers it’s interested in, and performing its function. >> > >> >> > >> > >> >> > Honestly, this discussion has been going on for a long time, and >> > it’s >> > >> >> > always “Oh, you came up with 2 use cases, and yeah, those use >> cases >> > >> are >> > >> >> > real things that someone would want to do. Here’s an alternate >> way >> > to >> > >> >> > implement them so let’s not do headers.” If we have a few use >> cases >> > >> that >> > >> >> we >> > >> >> > actually came up with, you can be sure that over the next year >> > >> there’s a >> > >> >> > dozen others that we didn’t think of that someone would like to >> > do. I >> > >> >> > really think it’s time to stop rehashing this discussion and >> > instead >> > >> >> focus >> > >> >> > on a workable standard that we can adopt. >> > >> >> > >> > >> >> > -Todd >> > >> >> > >> > >> >> > >> > >> >> > On Thu, Dec 1, 2016 at 1:39 PM, Todd Palino <tpal...@gmail.com> >> > >> wrote: >> > >> >> > >> > >> >> >> C. per message encryption >> > >> >> >>> One drawback of this approach is that this significantly reduce >> > the >> > >> >> >>> effectiveness of compression, which happens on a set of >> > serialized >> > >> >> >>> messages. An alternative is to enable SSL for wire encryption >> and >> > >> rely >> > >> >> on >> > >> >> >>> the storage system (e.g. LUKS) for at rest encryption. >> > >> >> >> >> > >> >> >> >> > >> >> >> Jun, this is not sufficient. While this does cover the case of >> > >> removing >> > >> >> a >> > >> >> >> drive from the system, it will not satisfy most compliance >> > >> requirements >> > >> >> for >> > >> >> >> encryption of data as whoever has access to the broker itself >> > still >> > >> has >> > >> >> >> access to the unencrypted data. For end-to-end encryption you >> > need to >> > >> >> >> encrypt at the producer, before it enters the system, and >> decrypt >> > at >> > >> the >> > >> >> >> consumer, after it exits the system. >> > >> >> >> >> > >> >> >> -Todd >> > >> >> >> >> > >> >> >> >> > >> >> >> On Thu, Dec 1, 2016 at 1:03 PM, radai < >> radai.rosenbl...@gmail.com >> > > >> > >> >> wrote: >> > >> >> >> >> > >> >> >>> another big plus of headers in the protocol is that it would >> > enable >> > >> >> rapid >> > >> >> >>> iteration on ideas outside of core kafka and would reduce the >> > >> number of >> > >> >> >>> future wire format changes required. >> > >> >> >>> >> > >> >> >>> a lot of what is currently a KIP represents use cases that are >> > not >> > >> 100% >> > >> >> >>> relevant to all users, and some of them require rather invasive >> > wire >> > >> >> >>> protocol changes. a thing a good recent example of this is >> > kip-98. >> > >> >> >>> tx-utilizing traffic is expected to be a very small fraction of >> > >> total >> > >> >> >>> traffic and yet the changes are invasive. >> > >> >> >>> >> > >> >> >>> every such wire format change translates into painful and slow >> > >> >> adoption of >> > >> >> >>> new versions. >> > >> >> >>> >> > >> >> >>> i think a lot of functionality currently in KIPs could be "spun >> > out" >> > >> >> and >> > >> >> >>> implemented as opt-in plugins transmitting data over headers. >> > this >> > >> >> would >> > >> >> >>> keep the core wire format stable(r), core codebase smaller, and >> > >> avoid >> > >> >> the >> > >> >> >>> "burden of proof" thats sometimes required to prove a certain >> > >> feature >> > >> >> is >> > >> >> >>> useful enough for a wide-enough audience to warrant a wire >> format >> > >> >> change >> > >> >> >>> and code complexity additions. >> > >> >> >>> >> > >> >> >>> (to be clear - kip-98 goes beyond "mere" wire format changes >> and >> > im >> > >> not >> > >> >> >>> saying it could have been completely done with headers, but >> > >> >> exactly-once >> > >> >> >>> delivery certainly could) >> > >> >> >>> >> > >> >> >>> On Thu, Dec 1, 2016 at 11:20 AM, Gwen Shapira < >> g...@confluent.io >> > > >> > >> >> wrote: >> > >> >> >>> >> > >> >> >>> > On Thu, Dec 1, 2016 at 10:24 AM, radai < >> > >> radai.rosenbl...@gmail.com> >> > >> >> >>> wrote: >> > >> >> >>> > > "For use cases within an organization, one could always use >> > >> other >> > >> >> >>> > > approaches such as company-wise containers" >> > >> >> >>> > > this is what linkedin has traditionally done but there are >> > now >> > >> >> cases >> > >> >> >>> > (read >> > >> >> >>> > > - topics) where this is not acceptable. this makes headers >> > >> useful >> > >> >> even >> > >> >> >>> > > within single orgs for cases where one-container-fits-all >> > cannot >> > >> >> >>> apply. >> > >> >> >>> > > >> > >> >> >>> > > as for the particular use cases listed, i dont want this to >> > >> devolve >> > >> >> >>> to a >> > >> >> >>> > > discussion of particular use cases - i think its enough >> that >> > >> some >> > >> >> of >> > >> >> >>> them >> > >> >> >>> > >> > >> >> >>> > I think a main point of contention is that: We identified few >> > >> >> >>> > use-cases where headers are useful, do we want Kafka to be a >> > >> system >> > >> >> >>> > that supports those use-cases? >> > >> >> >>> > >> > >> >> >>> > For example, Jun said: >> > >> >> >>> > "Not sure how widely useful record-level lineage is though >> > since >> > >> the >> > >> >> >>> > overhead could >> > >> >> >>> > be significant." >> > >> >> >>> > >> > >> >> >>> > We know NiFi supports record level lineage. I don't think it >> > was >> > >> >> >>> > developed for lols, I think it is safe to assume that the NSA >> > >> needed >> > >> >> >>> > that functionality. We also know that certain financial >> > institutes >> > >> >> >>> > need to track tampering with records at a record level and >> > there >> > >> are >> > >> >> >>> > federal regulations that absolutely require this. They also >> > need >> > >> to >> > >> >> >>> > prove that routing apps that "touches" the messages and >> either >> > >> reads >> > >> >> >>> > or updates headers couldn't have possibly modified the >> payload >> > >> >> itself. >> > >> >> >>> > They use record level encryption to do that - apps can read >> and >> > >> >> >>> > (sometimes) modify headers but can't touch the payload. >> > >> >> >>> > >> > >> >> >>> > We can totally say "those are corner cases and not worth >> adding >> > >> >> >>> > headers to Kafka for", they should use a different pubsub >> > message >> > >> for >> > >> >> >>> > that (Nifi or one of the other 1000 that cater specifically >> to >> > the >> > >> >> >>> > financial industry). >> > >> >> >>> > >> > >> >> >>> > But this gets us into a catch 22: >> > >> >> >>> > If we discuss a specific use-case, someone can always say it >> > isn't >> > >> >> >>> > interesting enough for Kafka. If we discuss more general >> > trends, >> > >> >> >>> > others can say "well, we are not sure any of them really >> needs >> > >> >> headers >> > >> >> >>> > specifically. This is just hand waving and not interesting.". >> > >> >> >>> > >> > >> >> >>> > I think discussing use-cases in specifics is super important >> to >> > >> >> decide >> > >> >> >>> > implementation details for headers (my use-cases lean toward >> > >> >> numerical >> > >> >> >>> > keys with namespaces and object values, others differ), but I >> > >> think >> > >> >> we >> > >> >> >>> > need to answer the general "Are we going to have headers" >> > question >> > >> >> >>> > first. >> > >> >> >>> > >> > >> >> >>> > I'd love to hear from the other committers in the discussion: >> > >> >> >>> > What would it take to convince you that headers in Kafka are >> a >> > >> good >> > >> >> >>> > idea in general, so we can move ahead and try to agree on the >> > >> >> details? >> > >> >> >>> > >> > >> >> >>> > I feel like we keep moving the goal posts and this is truly >> > >> >> exhausting. >> > >> >> >>> > >> > >> >> >>> > For the record, I mildly support adding headers to Kafka >> > (+0.5?). >> > >> >> >>> > The community can continue to find workarounds to the issue >> and >> > >> there >> > >> >> >>> > are some benefits to keeping the message format and clients >> > >> simpler. >> > >> >> >>> > But I see the usefulness of headers to many use-cases and if >> we >> > >> can >> > >> >> >>> > find a good and generally useful way to add it to Kafka, it >> > will >> > >> make >> > >> >> >>> > Kafka easier to use for many - worthy goal in my eyes. >> > >> >> >>> > >> > >> >> >>> > > are interesting/feasible, but: >> > >> >> >>> > > A+B. i think there are use cases for polyglot topics. >> > >> especially if >> > >> >> >>> kafka >> > >> >> >>> > > is being used to "trunk" something else. >> > >> >> >>> > > D. multiple topics would make it harder to write portable >> > >> consumer >> > >> >> >>> code. >> > >> >> >>> > > partition remapping would mess with locality of consumption >> > >> >> >>> guarantees. >> > >> >> >>> > > E+F. a use case I see for lineage/metadata is >> > >> billing/chargeback. >> > >> >> for >> > >> >> >>> > that >> > >> >> >>> > > use case it is not enough to simply record the point of >> > origin, >> > >> but >> > >> >> >>> every >> > >> >> >>> > > replication stop (think mirror maker) must also add a >> record >> > to >> > >> >> form a >> > >> >> >>> > > "transit log". >> > >> >> >>> > > >> > >> >> >>> > > as for stream processing on top of kafka - i know samza >> has a >> > >> >> metadata >> > >> >> >>> > map >> > >> >> >>> > > which they carry around in addition to user values. headers >> > are >> > >> the >> > >> >> >>> > perfect >> > >> >> >>> > > fit for these things. >> > >> >> >>> > > >> > >> >> >>> > > >> > >> >> >>> > > >> > >> >> >>> > > On Wed, Nov 30, 2016 at 6:50 PM, Jun Rao <j...@confluent.io >> > >> > >> wrote: >> > >> >> >>> > > >> > >> >> >>> > >> Hi, Michael, >> > >> >> >>> > >> >> > >> >> >>> > >> In order to answer the first two questions, it would be >> > helpful >> > >> >> if we >> > >> >> >>> > could >> > >> >> >>> > >> identify 1 or 2 strong use cases for headers in the space >> > for >> > >> >> >>> > third-party >> > >> >> >>> > >> vendors. For use cases within an organization, one could >> > always >> > >> >> use >> > >> >> >>> > other >> > >> >> >>> > >> approaches such as company-wise containers to get around >> w/o >> > >> >> >>> headers. I >> > >> >> >>> > >> went through the use cases in the KIP and in Radai's wiki >> ( >> > >> >> >>> > >> https://cwiki.apache.org/confluence/display/KAFKA/A+ >> > >> >> >>> > Case+for+Kafka+Headers >> > >> >> >>> > >> ). >> > >> >> >>> > >> The following are the ones that that I understand and >> could >> > be >> > >> in >> > >> >> the >> > >> >> >>> > >> third-party use case category. >> > >> >> >>> > >> >> > >> >> >>> > >> A. content-type >> > >> >> >>> > >> It seems that in general, content-type should be set at >> the >> > >> topic >> > >> >> >>> level. >> > >> >> >>> > >> Not sure if mixing messages with different content types >> > >> should be >> > >> >> >>> > >> encouraged. >> > >> >> >>> > >> >> > >> >> >>> > >> B. schema id >> > >> >> >>> > >> Since the value is mostly useless without schema id, it >> > seems >> > >> that >> > >> >> >>> > storing >> > >> >> >>> > >> the schema id together with serialized bytes in the value >> is >> > >> >> better? >> > >> >> >>> > >> >> > >> >> >>> > >> C. per message encryption >> > >> >> >>> > >> One drawback of this approach is that this significantly >> > reduce >> > >> >> the >> > >> >> >>> > >> effectiveness of compression, which happens on a set of >> > >> serialized >> > >> >> >>> > >> messages. An alternative is to enable SSL for wire >> > encryption >> > >> and >> > >> >> >>> rely >> > >> >> >>> > on >> > >> >> >>> > >> the storage system (e.g. LUKS) for at rest encryption. >> > >> >> >>> > >> >> > >> >> >>> > >> D. cluster ID for mirroring across Kafka clusters >> > >> >> >>> > >> This is actually interesting. Today, to avoid introducing >> > >> cycles >> > >> >> when >> > >> >> >>> > doing >> > >> >> >>> > >> mirroring across data centers, one would either have to >> set >> > up >> > >> two >> > >> >> >>> Kafka >> > >> >> >>> > >> clusters (a local and an aggregate) per data center or >> > rename >> > >> >> topics. >> > >> >> >>> > >> Neither is ideal. With headers, the producer could tag >> each >> > >> >> message >> > >> >> >>> with >> > >> >> >>> > >> the producing cluster ID in the header. MirrorMaker could >> > then >> > >> >> avoid >> > >> >> >>> > >> mirroring messages to a cluster if they are tagged with >> the >> > >> same >> > >> >> >>> cluster >> > >> >> >>> > >> id. >> > >> >> >>> > >> >> > >> >> >>> > >> However, an alternative approach is to introduce sth like >> > >> >> >>> hierarchical >> > >> >> >>> > >> topic and store messages from different clusters in >> > different >> > >> >> >>> partitions >> > >> >> >>> > >> under the same topic. This approach avoids filtering out >> > >> unneeded >> > >> >> >>> data >> > >> >> >>> > and >> > >> >> >>> > >> makes offset preserving easier to support. It may make >> > >> compaction >> > >> >> >>> > trickier >> > >> >> >>> > >> though since the same key may show up in different >> > partitions. >> > >> >> >>> > >> >> > >> >> >>> > >> E. record-level lineage >> > >> >> >>> > >> For example, a source connector could store in the message >> > the >> > >> >> >>> metadata >> > >> >> >>> > >> (e.g. UUID) of the source record. Similarly, if a stream >> job >> > >> >> >>> transforms >> > >> >> >>> > >> messages from topic A to topic B, the library could >> include >> > the >> > >> >> >>> source >> > >> >> >>> > >> message offset in each of the transformed message in the >> > >> header. >> > >> >> Not >> > >> >> >>> > sure >> > >> >> >>> > >> how widely useful record-level lineage is though since the >> > >> >> overhead >> > >> >> >>> > could >> > >> >> >>> > >> be significant. >> > >> >> >>> > >> >> > >> >> >>> > >> F. auditing metadata >> > >> >> >>> > >> We could put things like clientId/host/user in the header >> in >> > >> each >> > >> >> >>> > message >> > >> >> >>> > >> for auditing. These metadata are really at the producer >> > level >> > >> >> though. >> > >> >> >>> > So, a >> > >> >> >>> > >> more efficient way is to only include a "producerId" per >> > >> message >> > >> >> and >> > >> >> >>> > send >> > >> >> >>> > >> the producerId -> metadata mapping independently. KIP-98 >> is >> > >> >> actually >> > >> >> >>> > >> proposing including such a producerId natively in the >> > message. >> > >> >> >>> > >> >> > >> >> >>> > >> So, overall, I not sure that I am fully convinced of the >> > strong >> > >> >> >>> > third-party >> > >> >> >>> > >> use cases of headers yet. Perhaps we could discuss a bit >> > more >> > >> to >> > >> >> make >> > >> >> >>> > one >> > >> >> >>> > >> or two really convincing use cases. >> > >> >> >>> > >> >> > >> >> >>> > >> Another orthogonal question is whether header should be >> > >> exposed >> > >> >> in >> > >> >> >>> > stream >> > >> >> >>> > >> processing systems such Kafka stream, Samza, and Spark >> > >> streaming. >> > >> >> >>> > >> Currently, those systems just deal with key/value pairs. >> > >> Should we >> > >> >> >>> > expose a >> > >> >> >>> > >> third thing header there too or somehow map header to key >> or >> > >> >> value? >> > >> >> >>> > >> >> > >> >> >>> > >> Thanks, >> > >> >> >>> > >> >> > >> >> >>> > >> Jun >> > >> >> >>> > >> >> > >> >> >>> > >> >> > >> >> >>> > >> On Tue, Nov 29, 2016 at 3:35 AM, Michael Pearce < >> > >> >> >>> michael.pea...@ig.com> >> > >> >> >>> > >> wrote: >> > >> >> >>> > >> >> > >> >> >>> > >> > I assume, that after a period of a week, that there is >> no >> > >> >> concerns >> > >> >> >>> now >> > >> >> >>> > >> > with points 1, and 2 and now we have agreement that >> > headers >> > >> are >> > >> >> >>> useful >> > >> >> >>> > >> and >> > >> >> >>> > >> > needed in Kafka. As such if put to a KIP vote, this >> > wouldn’t >> > >> be >> > >> >> a >> > >> >> >>> > reason >> > >> >> >>> > >> to >> > >> >> >>> > >> > reject. >> > >> >> >>> > >> > >> > >> >> >>> > >> > @ >> > >> >> >>> > >> > Ignacio on point 4). >> > >> >> >>> > >> > I think for purpose of getting this KIP moving past >> this, >> > we >> > >> can >> > >> >> >>> state >> > >> >> >>> > >> the >> > >> >> >>> > >> > key will be a 4 bytes space that can will be naturally >> > >> >> interpreted >> > >> >> >>> as >> > >> >> >>> > an >> > >> >> >>> > >> > Int32 (if namespacing is later wanted you can easily >> split >> > >> this >> > >> >> >>> into >> > >> >> >>> > two >> > >> >> >>> > >> > int16 spaces), from the wire protocol implementation >> this >> > >> makes >> > >> >> no >> > >> >> >>> > >> > difference I don’t believe. Is this reasonable to all? >> > >> >> >>> > >> > >> > >> >> >>> > >> > On 5) as per point 4 therefor happy we keep with 32 >> bits. >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > On 18/11/2016, 20:34, "ignacio.so...@gmail.com on >> behalf >> > of >> > >> >> >>> Ignacio >> > >> >> >>> > >> > Solis" <ignacio.so...@gmail.com on behalf of >> > iso...@igso.net >> > >> > >> > >> >> >>> wrote: >> > >> >> >>> > >> > >> > >> >> >>> > >> > Summary: >> > >> >> >>> > >> > >> > >> >> >>> > >> > 3) Yes - Header value as byte[] >> > >> >> >>> > >> > >> > >> >> >>> > >> > 4a) Int,Int - No >> > >> >> >>> > >> > 4b) Int - Yes >> > >> >> >>> > >> > 4c) String - Reluctant maybe >> > >> >> >>> > >> > >> > >> >> >>> > >> > 5) I believe the header system should take a single >> > >> int. I >> > >> >> >>> think >> > >> >> >>> > >> > 32bits is >> > >> >> >>> > >> > a good size, if you want to interpret this as to >> 16bit >> > >> >> numbers >> > >> >> >>> in >> > >> >> >>> > the >> > >> >> >>> > >> > layer >> > >> >> >>> > >> > above go right ahead. If somebody wants to argue >> for >> > 16 >> > >> >> bits >> > >> >> >>> or >> > >> >> >>> > 64 >> > >> >> >>> > >> > bits of >> > >> >> >>> > >> > header key space I would listen. >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > Discussion: >> > >> >> >>> > >> > Dividing the key space into sub_key_1 and sub_key_2 >> > >> makes no >> > >> >> >>> > sense to >> > >> >> >>> > >> > me at >> > >> >> >>> > >> > this layer. Are we going to start providing APIs to >> > get >> > >> all >> > >> >> >>> the >> > >> >> >>> > >> > sub_key_1s? or all the sub_key_2s? If there is no >> > >> >> >>> distinguishing >> > >> >> >>> > >> > functions >> > >> >> >>> > >> > that are applied to each one then they should be a >> > single >> > >> >> >>> value. >> > >> >> >>> > At >> > >> >> >>> > >> > this >> > >> >> >>> > >> > layer all we're doing is equality. >> > >> >> >>> > >> > If the above layer wants to interpret this as 2, 3 >> or >> > >> more >> > >> >> >>> values >> > >> >> >>> > >> > that's a >> > >> >> >>> > >> > different question. I personally think it's all one >> > >> >> keyspace >> > >> >> >>> > that is >> > >> >> >>> > >> > getting assigned using some structure, but if you >> > want to >> > >> >> >>> > sub-assign >> > >> >> >>> > >> > parts >> > >> >> >>> > >> > of it then that's fine. >> > >> >> >>> > >> > >> > >> >> >>> > >> > The same discussion applies to strings. If somebody >> > >> argued >> > >> >> for >> > >> >> >>> > >> > strings, >> > >> >> >>> > >> > would we be arguing to divide the strings with dots >> > ('.') >> > >> >> as a >> > >> >> >>> > >> > requirement? >> > >> >> >>> > >> > Would we want them to give us the different name >> > segments >> > >> >> >>> > separately? >> > >> >> >>> > >> > Would we be performing any actions on this key other >> > than >> > >> >> >>> > matching? >> > >> >> >>> > >> > >> > >> >> >>> > >> > Nacho >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > >> > >> >> >>> > >> > On Fri, Nov 18, 2016 at 9:30 AM, Michael Pearce < >> > >> >> >>> > >> michael.pea...@ig.com >> > >> >> >>> > >> > > >> > >> >> >>> > >> > wrote: >> > >> >> >>> > >> > >> > >> >> >>> > >> > > #jay #jun any concerns on 1 and 2 still? >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > @all >> > >> >> >>> > >> > > To get this moving along a bit more I'd also like >> to >> > >> ask >> > >> >> to >> > >> >> >>> get >> > >> >> >>> > >> > clarity on >> > >> >> >>> > >> > > the below last points: >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > 3) I believe we're all roughly happy with the >> header >> > >> value >> > >> >> >>> > being a >> > >> >> >>> > >> > byte[]? >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > 4) I believe consensus has been for an namespace >> > based >> > >> int >> > >> >> >>> > approach >> > >> >> >>> > >> > > {int,int} for the key. Any objections if this is >> > what >> > >> we >> > >> >> go >> > >> >> >>> > with? >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > 5) as we have if assumption in (4) is correct, >> > >> {int,int} >> > >> >> >>> keys. >> > >> >> >>> > >> > > Should both int's be int16 or int32? >> > >> >> >>> > >> > > I'm for them being int16(2 bytes) as combined is >> > space >> > >> of >> > >> >> >>> > 4bytes as >> > >> >> >>> > >> > per >> > >> >> >>> > >> > > original and gives plenty of combinations for the >> > >> >> >>> foreseeable, >> > >> >> >>> > and >> > >> >> >>> > >> > keeps >> > >> >> >>> > >> > > the overhead small. >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > Do we see any benefit in another kip call to >> discuss >> > >> >> these at >> > >> >> >>> > all? >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > Cheers >> > >> >> >>> > >> > > Mike >> > >> >> >>> > >> > > ________________________________________ >> > >> >> >>> > >> > > From: K Burstev <k.burs...@yandex.com> >> > >> >> >>> > >> > > Sent: Friday, November 18, 2016 7:07:07 AM >> > >> >> >>> > >> > > To: dev@kafka.apache.org >> > >> >> >>> > >> > > Subject: Re: [DISCUSS] KIP-82 - Add Record Headers >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > For what it is worth also i agree. As a user: >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > 1) Yes - Headers are worthwhile >> > >> >> >>> > >> > > 2) Yes - Headers should be a top level option >> > >> >> >>> > >> > > >> > >> >> >>> > >> > > 14.11.2016, 21:15, "Ignacio Solis" < >> iso...@igso.net >> > >: >> > >> >> >>> > >> > > > 1) Yes - Headers are worthwhile >> > >> >> >>> > >> > > > 2) Yes - Headers should be a top level option >> > >> >> >>> > >> > > > >> > >> >> >>> > >> > > > On Mon, Nov 14, 2016 at 9:16 AM, Michael Pearce >> < >> > >> >> >>> > >> > michael.pea...@ig.com> >> > >> >> >>> > >> > > > wrote: >> > >> >> >>> > >> > > > >> > >> >> >>> > >> > > >> Hi Roger, >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> The kip details/examples the original proposal >> > for >> > >> key >> > >> >> >>> > spacing >> > >> >> >>> > >> , >> > >> >> >>> > >> > not >> > >> >> >>> > >> > > the >> > >> >> >>> > >> > > >> new mentioned as per discussion namespace >> idea. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> We will need to update the kip, when we get >> > >> agreement >> > >> >> >>> this >> > >> >> >>> > is a >> > >> >> >>> > >> > better >> > >> >> >>> > >> > > >> approach (which seems to be the case if I have >> > >> >> understood >> > >> >> >>> > the >> > >> >> >>> > >> > general >> > >> >> >>> > >> > > >> feeling in the conversation) >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Re the variable ints, at very early stage we >> did >> > >> think >> > >> >> >>> about >> > >> >> >>> > >> > this. I >> > >> >> >>> > >> > > think >> > >> >> >>> > >> > > >> the added complexity for the saving isn't >> worth >> > it. >> > >> >> I'd >> > >> >> >>> > rather >> > >> >> >>> > >> go >> > >> >> >>> > >> > > with, if >> > >> >> >>> > >> > > >> we want to reduce overheads and size int16 >> > (2bytes) >> > >> >> keys >> > >> >> >>> as >> > >> >> >>> > it >> > >> >> >>> > >> > keeps it >> > >> >> >>> > >> > > >> simple. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> On the note of no headers, there is as per the >> > kip >> > >> as >> > >> >> we >> > >> >> >>> > use an >> > >> >> >>> > >> > > attribute >> > >> >> >>> > >> > > >> bit to denote if headers are present or not as >> > such >> > >> >> >>> > provides a >> > >> >> >>> > >> > zero >> > >> >> >>> > >> > > >> overhead currently if headers are not used. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> I think as radai mentions would be good first >> > if we >> > >> >> can >> > >> >> >>> get >> > >> >> >>> > >> > clarity if >> > >> >> >>> > >> > > do >> > >> >> >>> > >> > > >> we now have general consensus that (1) headers >> > are >> > >> >> >>> > worthwhile >> > >> >> >>> > >> and >> > >> >> >>> > >> > > useful, >> > >> >> >>> > >> > > >> and (2) we want it as a top level entity. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Just to state the obvious i believe (1) >> headers >> > are >> > >> >> >>> > worthwhile >> > >> >> >>> > >> > and (2) >> > >> >> >>> > >> > > >> agree as a top level entity. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Cheers >> > >> >> >>> > >> > > >> Mike >> > >> >> >>> > >> > > >> ________________________________________ >> > >> >> >>> > >> > > >> From: Roger Hoover <roger.hoo...@gmail.com> >> > >> >> >>> > >> > > >> Sent: Wednesday, November 9, 2016 9:10:47 PM >> > >> >> >>> > >> > > >> To: dev@kafka.apache.org >> > >> >> >>> > >> > > >> Subject: Re: [DISCUSS] KIP-82 - Add Record >> > Headers >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Sorry for going a little in the weeds but >> thanks >> > >> for >> > >> >> the >> > >> >> >>> > >> replies >> > >> >> >>> > >> > > regarding >> > >> >> >>> > >> > > >> varint. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Agreed that a prefix and {int, int} can be the >> > >> same. >> > >> >> It >> > >> >> >>> > doesn't >> > >> >> >>> > >> > look >> > >> >> >>> > >> > > like >> > >> >> >>> > >> > > >> that's what the KIP is saying the "Open" >> > section. >> > >> The >> > >> >> >>> > example >> > >> >> >>> > >> > shows >> > >> >> >>> > >> > > >> 2100001 >> > >> >> >>> > >> > > >> for New Relic and 210002 for App Dynamics >> > implying >> > >> >> that >> > >> >> >>> the >> > >> >> >>> > New >> > >> >> >>> > >> > Relic >> > >> >> >>> > >> > > >> organization will have only a single header id >> > to >> > >> work >> > >> >> >>> > with. Or >> > >> >> >>> > >> > is >> > >> >> >>> > >> > > 2100001 >> > >> >> >>> > >> > > >> a prefix? The main point of a namespace or >> > prefix >> > >> is >> > >> >> to >> > >> >> >>> > reduce >> > >> >> >>> > >> > the >> > >> >> >>> > >> > > >> overhead of config mapping or registration >> > >> depending >> > >> >> on >> > >> >> >>> how >> > >> >> >>> > >> > > >> namespaces/prefixes are managed. >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Would love to hear more feedback on the >> > >> higher-level >> > >> >> >>> > questions >> > >> >> >>> > >> > > though... >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Cheers, >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> Roger >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> On Wed, Nov 9, 2016 at 11:38 AM, radai < >> > >> >> >>> > >> > radai.rosenbl...@gmail.com> >> > >> >> >>> > >> > > wrote: >> > >> >> >>> > >> > > >> >> > >> >> >>> > >> > > >> > I think this discussion is getting a bit >> into >> > the >> > >> >> >>> weeds on >> > >> >> >>> > >> > technical >> > >> >> >>> > >> > > >> > implementation details. >> > >> >> >>> > >> > > >> > I'd liek to step back a minute and try and >> > >> establish >> > >> >> >>> > where we >> > >> >> >>> > >> > are in >> > >> >> >>> > >> > > the >> > >> >> >>> > >> > > >> > larger picture: >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > (re-wording nacho's last paragraph) >> > >> >> >>> > >> > > >> > 1. are we all in agreement that headers are >> a >> > >> >> >>> worthwhile >> > >> >> >>> > and >> > >> >> >>> > >> > useful >> > >> >> >>> > >> > > >> > addition to have? this was contested early >> on >> > >> >> >>> > >> > > >> > 2. are we all in agreement on headers as top >> > >> level >> > >> >> >>> entity >> > >> >> >>> > vs >> > >> >> >>> > >> > headers >> > >> >> >>> > >> > > >> > squirreled-away in V? >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > if there are still concerns around these #2 >> > >> points >> > >> >> >>> (#jay? >> > >> >> >>> > >> > #jun?)? >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > (and now back to our normal programming ...) >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > varints are nice. having said that, its >> adding >> > >> >> >>> complexity >> > >> >> >>> > >> (see >> > >> >> >>> > >> > > >> > https://github.com/addthis/ >> > >> >> stream-lib/blob/master/src/ >> > >> >> >>> > >> > > >> > main/java/com/clearspring/ >> > >> >> analytics/util/Varint.java >> > >> >> >>> > >> > > >> > as 1st google result) and would require >> anyone >> > >> >> writing >> > >> >> >>> > other >> > >> >> >>> > >> > clients >> > >> >> >>> > >> > > (C? >> > >> >> >>> > >> > > >> > Python? Go? Bash? ;-) ) to get/implement the >> > >> same, >> > >> >> and >> > >> >> >>> for >> > >> >> >>> > >> > relatively >> > >> >> >>> > >> > > >> > little gain (int vs string is order of >> > magnitude, >> > >> >> this >> > >> >> >>> > isnt). >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > int namespacing vs {int, int} namespacing >> are >> > >> >> basically >> > >> >> >>> > the >> > >> >> >>> > >> > same >> > >> >> >>> > >> > > thing - >> > >> >> >>> > >> > > >> > youre just namespacing an int64 and giving >> > people >> > >> >> while >> > >> >> >>> > 2^32 >> > >> >> >>> > >> > ranges >> > >> >> >>> > >> > > at a >> > >> >> >>> > >> > > >> > time. the part i like about this is letting >> > >> people >> > >> >> >>> have a >> > >> >> >>> > >> large >> > >> >> >>> > >> > > swath of >> > >> >> >>> > >> > > >> > numbers with one registration so they dont >> > have >> > >> to >> > >> >> come >> > >> >> >>> > back >> > >> >> >>> > >> > for >> > >> >> >>> > >> > > every >> > >> >> >>> > >> > > >> > single plugin/header they want to "reserve". >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > On Wed, Nov 9, 2016 at 11:01 AM, Roger >> Hoover >> > < >> > >> >> >>> > >> > > roger.hoo...@gmail.com> >> > >> >> >>> > >> > > >> > wrote: >> > >> >> >>> > >> > > >> > >> > >> >> >>> > >> > > >> > > Since some of the debate has been about >> > >> overhead + >> > >> >> >>> > >> > performance, I'm >> > >> >> >>> > >> > > >> > > wondering if we have considered a varint >> > >> encoding >> > >> >> ( >> > >> >> >>> > >> > > >> > > https://developers.google.com/ >> > >> >> protocol-buffers/docs/ >> > >> >> >>> > >> > > encoding#varints) >> > >> >> >>> > >> > > >> > for >> > >> >> >>> > >> > > >> > > the header length field (int32 in the >> > proposal) >> > >> >> and >> > >> >> >>> for >> > >> >> >>> > >> > header >> > >> >> >>> > >> > > ids? If >> > >> >> >>> > >> > > >> > you >> > >> >> >>> > >> > > >> > > don't use headers, the overhead would be a >> > >> single >> > >> >> >>> byte >> > >> >> >>> > and >> > >> >> >>> > >> > for each >> > >> >> >>> > >> > > >> > header >> > >> >> >>> > >> > > >> > > id < 128 would also need only a single >> byte? >> > >> >> >>> > >> > > >> > > >> > >> >> >>> > >> > > >> > > >> > >> >> >>> > >> > > >> > > >> > >> >> >>> > >> > > >> > > On Wed, Nov 9, 2016 at 6:43 AM, radai < >> > >> >> >>> > >> > radai.rosenbl...@gmail.com> >> > >> >> >>> > >> > > >> > wrote: >> > >> >> >>> > >> > > >> > > >> > >> >> >>> > >> > > >> > > > @magnus - and very dangerous (youre >> > >> essentially >> > >> >> >>> > >> > downloading and >> > >> >> >>> > >> > > >> > executing >> > >> >> >>> > >> > > >> > > > arbitrary code off the internet on your >> > >> servers >> > >> >> ... >> > >> >> >>> > bad >> > >> >> >>> > >> > idea >> > >> >> >>> > >> > > without >> > >> >> >>> > >> > > >> a >> > >> >> >>> > >> > > >> > > > sandbox, even with) >> > >> >> >>> > >> > > >> > > > >> > >> >> >>> > >> > > >> > > > as for it being a purely administrative >> > task >> > >> - i >> > >> >> >>> > >> disagree. >> > >> >> >>> > >> > > >> > > > >> > >> >> >>> > >> > > >> > > > i wish it would, really, because then my >> > >> earlier >> > >> >> >>> > point on >> > >> >> >>> > >> > the >> > >> >> >>> > >> > > >> > complexity >> > >> >> >>> > >> > > >> > > of >> > >> >> >>> > >> > > >> > > > the remapping process would be invalid, >> > but >> > >> at >> > >> >> >>> > linkedin, >> > >> >> >>> > >> > for >> > >> >> >>> > >> > > example, >> > >> >> >>> > >> > > >> > we >> > >> >> >>> > >> > > >> > > > (the team im in) run kafka as a service. >> > we >> > >> dont >> > >> >> >>> > really >> > >> >> >>> > >> > know >> > >> >> >>> > >> > > what our >> > >> >> >>> > >> > > >> > > users >> > >> >> >>> > >> > > >> > > > (developing applications that use kafka) >> > are >> > >> up >> > >> >> to >> > >> >> >>> at >> > >> >> >>> > any >> > >> >> >>> > >> > given >> > >> >> >>> > >> > > >> moment. >> > >> >> >>> > >> > > >> > > it >> > >> >> >>> > >> > > >> > > > is very possible (given the existance of >> > >> headers >> > >> >> >>> and a >> > >> >> >>> > >> > > corresponding >> > >> >> >>> > >> > > >> > > plugin >> > >> >> >>> > >> > > >> > > > ecosystem) for some application to >> "equip" >> > >> their >> > >> >> >>> > >> producers >> > >> >> >>> > >> > and >> > >> >> >>> > >> > > >> > consumers >> > >> >> >>> > >> > > >> > > > with the required plugin without us >> > knowing. >> > >> i >> > >> >> dont >> > >> >> >>> > mean >> > >> >> >>> > >> > to imply >> > >> >> >>> > >> > > >> thats >> > >> >> >>> > >> > > >> > > > bad, i just want to make the point that >> > its >> > >> not >> > >> >> as >> > >> >> >>> > simple >> > >> >> >>> > >> > > keeping it >> > >> >> >>> > >> > > >> in >> > >> >> >>> > >> > > >> > > > sync across a large-enough organization. >> > >> >> >>> > >> > > >> > > > >> > >> >> >>> > >> > > >> > > > >> > >> >> >>> > >> > > >> > > > On Wed, Nov 9, 2016 at 6:17 AM, Magnus >> > >> Edenhill >> > >> >> < >> > >> >> >>> > >> > > mag...@edenhill.se> >> > >> >> >>> > >> > > >> > > > wrote: >> > >> >> >>> > >> > > >> > > > >> > >> >> >>> > >> > > >> > > > > I think there is a piece missing in >> the >> > >> >> Strings >> > >> >> >>> > >> > discussion, >> > >> >> >>> > >> > > where >> > >> >> >>> > >> > > >> > > > > pro-Stringers >> > >> >> >>> > >> > > >> > > > > reason that by providing unique string >> > >> >> >>> identifiers >> > >> >> >>> > for >> > >> >> >>> > >> > each >> > >> >> >>> > >> > > header >> > >> >> >>> > >> > > >> > > > > everything will just >> > >> >> >>> > >> > > >> > > > > magically work for all parts of the >> > stream >> > >> >> >>> pipeline. >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > But the strings dont mean anything by >> > >> >> themselves, >> > >> >> >>> > and >> > >> >> >>> > >> > while we >> > >> >> >>> > >> > > >> could >> > >> >> >>> > >> > > >> > > > > probably envision >> > >> >> >>> > >> > > >> > > > > some auto plugin loader that >> downloads, >> > >> >> compiles, >> > >> >> >>> > links >> > >> >> >>> > >> > and >> > >> >> >>> > >> > > runs >> > >> >> >>> > >> > > >> > > plugins >> > >> >> >>> > >> > > >> > > > > on-demand >> > >> >> >>> > >> > > >> > > > > as soon as they're seen by a >> consumer, I >> > >> dont >> > >> >> >>> really >> > >> >> >>> > >> see >> > >> >> >>> > >> > a >> > >> >> >>> > >> > > use-case >> > >> >> >>> > >> > > >> > for >> > >> >> >>> > >> > > >> > > > > something >> > >> >> >>> > >> > > >> > > > > so dynamic (and fragile) in practice. >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > In the real world an application will >> be >> > >> >> >>> configured >> > >> >> >>> > >> with >> > >> >> >>> > >> > a set >> > >> >> >>> > >> > > of >> > >> >> >>> > >> > > >> > > plugins >> > >> >> >>> > >> > > >> > > > > to either add (producer) >> > >> >> >>> > >> > > >> > > > > or read (consumer) headers. >> > >> >> >>> > >> > > >> > > > > This is an administrative task based >> on >> > >> what >> > >> >> >>> > features a >> > >> >> >>> > >> > client >> > >> >> >>> > >> > > >> > > > > needs/provides and results in >> > >> >> >>> > >> > > >> > > > > some sort of configuration to enable >> and >> > >> >> >>> configure >> > >> >> >>> > the >> > >> >> >>> > >> > desired >> > >> >> >>> > >> > > >> > plugins. >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > Since this needs to be kept somewhat >> in >> > >> sync >> > >> >> >>> across >> > >> >> >>> > an >> > >> >> >>> > >> > > organisation >> > >> >> >>> > >> > > >> > > > (there >> > >> >> >>> > >> > > >> > > > > is no point in having producers >> > >> >> >>> > >> > > >> > > > > add headers no consumers will read, >> and >> > >> vice >> > >> >> >>> versa), >> > >> >> >>> > >> the >> > >> >> >>> > >> > added >> > >> >> >>> > >> > > >> > > complexity >> > >> >> >>> > >> > > >> > > > > of assigning an id namespace >> > >> >> >>> > >> > > >> > > > > for each plugin as it is being >> > configured >> > >> >> should >> > >> >> >>> be >> > >> >> >>> > >> > tolerable. >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > /Magnus >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > 2016-11-09 13:06 GMT+01:00 Michael >> > Pearce < >> > >> >> >>> > >> > > michael.pea...@ig.com>: >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > > Just following/catching up on what >> > seems >> > >> to >> > >> >> be >> > >> >> >>> an >> > >> >> >>> > >> > active >> > >> >> >>> > >> > > night :) >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > @Radai sorry if it may seem obvious >> > but >> > >> what >> > >> >> >>> does >> > >> >> >>> > MD >> > >> >> >>> > >> > stand >> > >> >> >>> > >> > > for? >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > My take on String vs Int: >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > I will state first I am pro Int (16 >> or >> > >> 32). >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > I do though playing devils advocate >> > see a >> > >> >> big >> > >> >> >>> plus >> > >> >> >>> > >> > with the >> > >> >> >>> > >> > > >> > argument >> > >> >> >>> > >> > > >> > > of >> > >> >> >>> > >> > > >> > > > > > String keys, this is around >> > integrating >> > >> >> into an >> > >> >> >>> > >> > existing >> > >> >> >>> > >> > > >> > eco-system. >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > As many other systems use String >> based >> > >> >> headers >> > >> >> >>> > >> (Flume, >> > >> >> >>> > >> > JMS) >> > >> >> >>> > >> > > it >> > >> >> >>> > >> > > >> > makes >> > >> >> >>> > >> > > >> > > > it >> > >> >> >>> > >> > > >> > > > > > much easier for these to be >> > >> >> >>> > incorporated/integrated >> > >> >> >>> > >> > into. >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > How with Int based headers could we >> > >> provide >> > >> >> a >> > >> >> >>> > >> > way/guidence to >> > >> >> >>> > >> > > >> make >> > >> >> >>> > >> > > >> > > this >> > >> >> >>> > >> > > >> > > > > > integration simple / easy with >> > transition >> > >> >> flows >> > >> >> >>> > over >> > >> >> >>> > >> to >> > >> >> >>> > >> > > kafka? >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > * tough luck buddy you're on your >> own >> > >> >> >>> > >> > > >> > > > > > * simply hash the string into int >> code >> > >> and >> > >> >> hope >> > >> >> >>> > for >> > >> >> >>> > >> no >> > >> >> >>> > >> > > collisions >> > >> >> >>> > >> > > >> > > (how >> > >> >> >>> > >> > > >> > > > to >> > >> >> >>> > >> > > >> > > > > > convert back though?) >> > >> >> >>> > >> > > >> > > > > > * http2 style as mentioned by nacho. >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > cheers, >> > >> >> >>> > >> > > >> > > > > > Mike >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > ______________________________ >> > __________ >> > >> >> >>> > >> > > >> > > > > > From: radai < >> > radai.rosenbl...@gmail.com> >> > >> >> >>> > >> > > >> > > > > > Sent: Wednesday, November 9, 2016 >> > 8:12 AM >> > >> >> >>> > >> > > >> > > > > > To: dev@kafka.apache.org >> > >> >> >>> > >> > > >> > > > > > Subject: Re: [DISCUSS] KIP-82 - Add >> > >> Record >> > >> >> >>> Headers >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > thinking about it some more, the >> best >> > >> way to >> > >> >> >>> > transmit >> > >> >> >>> > >> > the >> > >> >> >>> > >> > > header >> > >> >> >>> > >> > > >> > > > > remapping >> > >> >> >>> > >> > > >> > > > > > data to consumers would be to put it >> > in >> > >> the >> > >> >> MD >> > >> >> >>> > >> response >> > >> >> >>> > >> > > payload, >> > >> >> >>> > >> > > >> so >> > >> >> >>> > >> > > >> > > > maybe >> > >> >> >>> > >> > > >> > > > > > it should be discussed now. >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > On Wed, Nov 9, 2016 at 12:09 AM, >> > radai < >> > >> >> >>> > >> > > >> radai.rosenbl...@gmail.com >> > >> >> >>> > >> > > >> > > >> > >> >> >>> > >> > > >> > > > > wrote: >> > >> >> >>> > >> > > >> > > > > > >> > >> >> >>> > >> > > >> > > > > > > im not opposed to the idea of >> > namespace >> > >> >> >>> mapping. >> > >> >> >>> > >> all >> > >> >> >>> > >> > im >> > >> >> >>> > >> > > saying >> > >> >> >>> > >> > > >> is >> > >> >> >>> > >> > > >> > > > that >> > >> >> >>> > >> > > >> > > > > > its >> > >> >> >>> > >> > > >> > > > > > > not part of the "mvp" and, since >> it >> > >> >> requires >> > >> >> >>> no >> > >> >> >>> > >> wire >> > >> >> >>> > >> > format >> > >> >> >>> > >> > > >> > change, >> > >> >> >>> > >> > > >> > > > can >> > >> >> >>> > >> > > >> > > > > > > always be added later. >> > >> >> >>> > >> > > >> > > > > > > also, its not as simple as just >> > >> >> configuring >> > >> >> >>> MM >> > >> >> >>> > to >> > >> >> >>> > >> do >> > >> >> >>> > >> > the >> > >> >> >>> > >> > > >> > transform: >> > >> >> >>> > >> > > >> > > > > lets >> > >> >> >>> > >> > > >> > > > > > > say i've implemented large message >> > >> >> support as >> > >> >> >>> > >> > {666,1} and >> > >> >> >>> > >> > > on >> > >> >> >>> > >> > > >> some >> > >> >> >>> > >> > > >> > > > > mirror >> > >> >> >>> > >> > > >> > > > > > > target cluster its been remapped >> to >> > >> >> {999,1}. >> > >> >> >>> the >> > >> >> >>> > >> > consumer >> > >> >> >>> > >> > > >> plugin >> > >> >> >>> > >> > > >> > > code >> > >> >> >>> > >> > > >> > > > > > would >> > >> >> >>> > >> > > >> > > > > > > also need to be told to look for >> the >> > >> large >> > >> >> >>> > message >> > >> >> >>> > >> > "part X >> > >> >> >>> > >> > > of >> > >> >> >>> > >> > > >> Y" >> > >> >> >>> > >> > > >> > > > header >> > >> >> >>> > >> > > >> > > > > > > under {999,1}. doable, but tricky. >> > >> >> >>> > >> > > >> > > > > > > >> > >> >> >>> > >> > > >> > > > > > > On Tue, Nov 8, 2016 at 10:29 PM, >> > Gwen >> > >> >> >>> Shapira < >> > >> >> >>> > >> > > >> g...@confluent.io >> > >> >> >>> > >> > > >> > > >> > >> >> >>> > >> > > >> > > > > wrote: >> > >> >> >>> > >> > > >> > > > > > > >> > >> >> >>> > >> > > >> > > > > > >> While you can do whatever you >> want >> > >> with a >> > >> >> >>> > >> namespace >> > >> >> >>> > >> > and >> > >> >> >>> > >> > > your >> > >> >> >>> > >> > > >> > code, >> > >> >> >>> > >> > > >> > > > > > >> what I'd expect is for each app >> to >> > >> >> >>> namespaces >> > >> >> >>> > >> > > configurable... >> > >> >> >>> > >> > > >> > > > > > >> >> > >> >> >>> > >> > > >> > > > > > >> So if I accidentally used 666 for >> > my >> > >> HR >> > >> >> >>> > >> department, >> > >> >> >>> > >> > and >> > >> >> >>> > >> > > still >> > >> >> >>> > >> > > >> > want >> > >> >> >>> > >> > > >> > > > to >> > >> >> >>> > >> > > >> > > > > > >> run RadaiApp, I can config >> > >> "namespace=42" >> > >> >> >>> for >> > >> >> >>> > >> > RadaiApp and >> > >> >> >>> > >> > > >> > > > everything >> > >> >> >>> > >> > > >> > > > > > >> will look normal. >> > >> >> >>> > >> > > >> > > > > > >> >> > >> >> >>> > >> > > >> > > > > > >> This means you only need to sync >> > usage >> > >> >> >>> inside >> > >> >> >>> > your >> > >> >> >>> > >> > own >> > >> >> >>> > >> > > >> > > organization. >> > >> >> >>> > >> > > >> > > > > > >> Still hard, but somewhat easier >> > than >> > >> >> syncing >> > >> >> >>> > with >> > >> >> >>> > >> > the >> > >> >> >>> > >> > > entire >> > >> >> >>> > >> > > >> > > world. >> > >> >> >>> > >> > > >> > > > > > >> >> > >> >> >>> > >> > > >> > > > > > >> On Tue, Nov 8, 2016 at 10:07 PM, >> > >> radai < >> > >> >> >>> > >> > > >> > > radai.rosenbl...@gmail.com> >> > >> >> >>> > >> > > >> > > > > > >> wrote: >> > >> >> >>> > >> > > >> > > > > > >> > and we can start with >> {namespace, >> > >> id} >> > >> >> and >> > >> >> >>> no >> > >> >> >>> > >> > re-mapping >> > >> >> >>> > >> > > >> > support >> > >> >> >>> > >> > > >> > > > and >> > >> >> >>> > >> > > >> > > > > > >> always >> > >> >> >>> > >> > > >> > > > > > >> > add it later on if/when >> > collisions >> > >> >> >>> actually >> > >> >> >>> > >> > happen (i >> > >> >> >>> > >> > > dont >> > >> >> >>> > >> > > >> > think >> > >> >> >>> > >> > > >> > > > > > they'd >> > >> >> >>> > >> > > >> > > > > > >> be >> > >> >> >>> > >> > > >> > > > > > >> > a problem). >> > >> >> >>> > >> > > >> > > > > > >> > >> > >> >> >>> > >> > > >> > > > > > >> > every interested party (so orgs >> > or >> > >> >> >>> > individuals) >> > >> >> >>> > >> > could >> > >> >> >>> > >> > > then >> > >> >> >>> > >> > > >> > > > register >> > >> >> >>> > >> > > >> > > > > a >> > >> >> >>> > >> > > >> > > > > > >> > prefix (0 = reserved, 1 = >> > confluent >> > >> ... >> > >> >> >>> 666 >> > >> >> >>> > = me >> > >> >> >>> > >> > :-) ) >> > >> >> >>> > >> > > and >> > >> >> >>> > >> > > >> do >> > >> >> >>> > >> > > >> > > > > whatever >> > >> >> >>> > >> > > >> > > > > > >> with >> > >> >> >>> > >> > > >> > > > > > >> > the 2nd ID - so once linkedin >> > >> >> registers, >> > >> >> >>> say >> > >> >> >>> > 3, >> > >> >> >>> > >> > then >> > >> >> >>> > >> > > >> linkedin >> > >> >> >>> > >> > > >> > > devs >> > >> >> >>> > >> > > >> > > > > are >> > >> >> >>> > >> > > >> > > > > > >> free >> > >> >> >>> > >> > > >> > > > > > >> > to use {3, *} with a reasonable >> > >> >> >>> expectation >> > >> >> >>> > to >> > >> >> >>> > >> to >> > >> >> >>> > >> > > collide >> > >> >> >>> > >> > > >> with >> > >> >> >>> > >> > > >> > > > > > anything >> > >> >> >>> > >> > > >> > > > > > >> > else. further partitioning of >> > that * >> > >> >> >>> becomes >> > >> >> >>> > >> > linkedin's >> > >> >> >>> > >> > > >> > problem, >> > >> >> >>> > >> > > >> > > > but >> > >> >> >>> > >> > > >> > > > > > the >> > >> >> >>> > >> > > >> > > > > > >> > "upstream registration" of a >> > >> namespace >> > >> >> >>> only >> > >> >> >>> > has >> > >> >> >>> > >> to >> > >> >> >>> > >> > > happen >> > >> >> >>> > >> > > >> > once. >> > >> >> >>> > >> > > >> > > > > > >> > >> > >> >> >>> > >> > > >> > > > > > >> > On Tue, Nov 8, 2016 at 9:03 PM, >> > >> James >> > >> >> >>> Cheng < >> > >> >> >>> > >> > > >> > > wushuja...@gmail.com >> > >> >> >>> > >> > > >> > > > > >> > >> >> >>> > >> > > >> > > > > > >> wrote: >> > >> >> >>> > >> > > >> > > > > > >> > >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> > On Nov 8, 2016, at 5:54 PM, >> > Gwen >> > >> >> >>> Shapira < >> > >> >> >>> > >> > > >> > g...@confluent.io> >> > >> >> >>> > >> > > >> > > > > > wrote: >> > >> >> >>> > >> > > >> > > > > > >> >> > >> > >> >> >>> > >> > > >> > > > > > >> >> > Thank you so much for this >> > clear >> > >> and >> > >> >> >>> fair >> > >> >> >>> > >> > summary of >> > >> >> >>> > >> > > the >> > >> >> >>> > >> > > >> > > > > arguments. >> > >> >> >>> > >> > > >> > > > > > >> >> > >> > >> >> >>> > >> > > >> > > > > > >> >> > I'm in favor of ints. Not a >> > >> >> >>> deal-breaker, >> > >> >> >>> > but >> > >> >> >>> > >> > in >> > >> >> >>> > >> > > favor. >> > >> >> >>> > >> > > >> > > > > > >> >> > >> > >> >> >>> > >> > > >> > > > > > >> >> > Even more in favor of >> Magnus's >> > >> >> >>> > decentralized >> > >> >> >>> > >> > > suggestion >> > >> >> >>> > >> > > >> > with >> > >> >> >>> > >> > > >> > > > > > Roger's >> > >> >> >>> > >> > > >> > > > > > >> >> > tweak: add a namespace for >> > >> headers. >> > >> >> >>> This >> > >> >> >>> > will >> > >> >> >>> > >> > allow >> > >> >> >>> > >> > > each >> > >> >> >>> > >> > > >> > app >> > >> >> >>> > >> > > >> > > to >> > >> >> >>> > >> > > >> > > > > > just >> > >> >> >>> > >> > > >> > > > > > >> >> > use whatever IDs it wants >> > >> >> internally, >> > >> >> >>> and >> > >> >> >>> > >> then >> > >> >> >>> > >> > let >> > >> >> >>> > >> > > the >> > >> >> >>> > >> > > >> > admin >> > >> >> >>> > >> > > >> > > > > > >> deploying >> > >> >> >>> > >> > > >> > > > > > >> >> > the app figure out an >> > available >> > >> >> >>> namespace >> > >> >> >>> > ID >> > >> >> >>> > >> > for the >> > >> >> >>> > >> > > app >> > >> >> >>> > >> > > >> to >> > >> >> >>> > >> > > >> > > > live >> > >> >> >>> > >> > > >> > > > > > in. >> > >> >> >>> > >> > > >> > > > > > >> >> > So >> > io.confluent.schema-registry >> > >> can >> > >> >> be >> > >> >> >>> > >> > namespace >> > >> >> >>> > >> > > 0x01 on >> > >> >> >>> > >> > > >> my >> > >> >> >>> > >> > > >> > > > > > >> deployment >> > >> >> >>> > >> > > >> > > > > > >> >> > and 0x57 on yours, and the >> > poor >> > >> guys >> > >> >> >>> > >> > developing the >> > >> >> >>> > >> > > app >> > >> >> >>> > >> > > >> > don't >> > >> >> >>> > >> > > >> > > > > need >> > >> >> >>> > >> > > >> > > > > > to >> > >> >> >>> > >> > > >> > > > > > >> >> > worry about that. >> > >> >> >>> > >> > > >> > > > > > >> >> > >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> Gwen, if I understand your >> > example >> > >> >> >>> right, an >> > >> >> >>> > >> > > application >> > >> >> >>> > >> > > >> > > deployer >> > >> >> >>> > >> > > >> > > > > > might >> > >> >> >>> > >> > > >> > > > > > >> >> decide to use 0x01 in one >> > >> deployment, >> > >> >> and >> > >> >> >>> > that >> > >> >> >>> > >> > means >> > >> >> >>> > >> > > that >> > >> >> >>> > >> > > >> > once >> > >> >> >>> > >> > > >> > > > the >> > >> >> >>> > >> > > >> > > > > > >> message >> > >> >> >>> > >> > > >> > > > > > >> >> is written into the broker, it >> > >> will be >> > >> >> >>> > saved on >> > >> >> >>> > >> > the >> > >> >> >>> > >> > > broker >> > >> >> >>> > >> > > >> > with >> > >> >> >>> > >> > > >> > > > > that >> > >> >> >>> > >> > > >> > > > > > >> >> specific namespace (0x01). >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> If you were to mirror that >> > message >> > >> >> into >> > >> >> >>> > another >> > >> >> >>> > >> > > cluster, >> > >> >> >>> > >> > > >> the >> > >> >> >>> > >> > > >> > > 0x01 >> > >> >> >>> > >> > > >> > > > > > would >> > >> >> >>> > >> > > >> > > > > > >> >> accompany the message, right? >> > What >> > >> if >> > >> >> the >> > >> >> >>> > >> > deployers of >> > >> >> >>> > >> > > the >> > >> >> >>> > >> > > >> > same >> > >> >> >>> > >> > > >> > > > app >> > >> >> >>> > >> > > >> > > > > > in >> > >> >> >>> > >> > > >> > > > > > >> the >> > >> >> >>> > >> > > >> > > > > > >> >> other cluster uses 0x57? They >> > won't >> > >> >> >>> > understand >> > >> >> >>> > >> > each >> > >> >> >>> > >> > > other? >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> I'm not sure that's an >> avoidable >> > >> >> >>> problem. I >> > >> >> >>> > >> > think it >> > >> >> >>> > >> > > simply >> > >> >> >>> > >> > > >> > > means >> > >> >> >>> > >> > > >> > > > > > that >> > >> >> >>> > >> > > >> > > > > > >> in >> > >> >> >>> > >> > > >> > > > > > >> >> order to share data, you have >> to >> > >> also >> > >> >> >>> have a >> > >> >> >>> > >> > shared >> > >> >> >>> > >> > > (agreed >> > >> >> >>> > >> > > >> > > upon) >> > >> >> >>> > >> > > >> > > > > > >> >> understanding of what the >> > >> namespaces >> > >> >> >>> mean. >> > >> >> >>> > >> Which >> > >> >> >>> > >> > I >> > >> >> >>> > >> > > think >> > >> >> >>> > >> > > >> > makes >> > >> >> >>> > >> > > >> > > > > sense, >> > >> >> >>> > >> > > >> > > > > > >> >> because the alternate (sharing >> > >> >> *nothing* >> > >> >> >>> at >> > >> >> >>> > >> all) >> > >> >> >>> > >> > would >> > >> >> >>> > >> > > mean >> > >> >> >>> > >> > > >> > > that >> > >> >> >>> > >> > > >> > > > > > there >> > >> >> >>> > >> > > >> > > > > > >> >> would be no way to understand >> > each >> > >> >> other. >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> -James >> > >> >> >>> > >> > > >> > > > > > >> >> >> > >> >> >>> > >> > > >> > > > > > >> >> > Gwen >> > >> >> >>> > >> > > >> > > > > > >> >> > >> > >> >> >>> > >> > > >> > > > > > >> >> > On Tue, Nov 8, 2016 at 4:23 >> > PM, >> > >> >> radai < >> > >> >> >>> > >> > > >> > > > > radai.rosenbl...@gmail.com> >> > >> >> >>> > >> > > >> > > > > > >> >> wrote: >> > >> >> >>> > >> > > >> > > > > > >> >> >> +1 for sean's document. it >> > >> covers >> > >> >> >>> pretty >> > >> >> >>> > >> much >> > >> >> >>> > >> > all >> > >> >> >>> > >> > > the >> > >> >> >>> > >> > > >> > > > trade-offs >> > >> >> >>> > >> > > >> > > > > > and >> > >> >> >>> > >> > > >> > > > > > >> >> >> provides concrete figures >> to >> > >> argue >> > >> >> >>> about >> > >> >> >>> > :-) >> > >> >> >>> > >> > > >> > > > > > >> >> >> (nit-picking - used the >> same >> > >> xkcd >> > >> >> >>> twice, >> > >> >> >>> > >> also >> > >> >> >>> > >> > trove >> > >> >> >>> > >> > > has >> > >> >> >>> > >> > > >> > been >> > >> >> >>> > >> > > >> > > > > > >> superceded >> > >> > >> > >> > >> > >> > -- >> > Gwen Shapira >> > Product Manager | Confluent >> > 650.450.2760 | @gwenshap >> > Follow us: Twitter | blog >> > >> >> >> >> -- >> *Todd Palino* >> Staff Site Reliability Engineer >> Data Infrastructure Streaming >> >> >> >> linkedin.com/in/toddpalino >> -- Gwen Shapira Product Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
