Thanks Guozhang and Matthias! I do also agree with this way of handling headers inheritance. I will add them to the KIP doc.
> We can discuss about extending the current protocol and how to enable users > override those rule, and how to expose them in the DSL layer in a future > KIP. About this, should this be managed on KIP-159 or a new one? El jue., 10 may. 2018 a las 17:46, Matthias J. Sax (<matth...@confluent.io>) escribió: > Thanks Guozhang! Sounds good to me! > > -Matthias > > On 5/10/18 7:55 AM, Guozhang Wang wrote: > > Thanks for your thoughts Matthias. I think if we do want to bring KIP-244 > > into 2.0 then we need to keep its scope small and well defined. For that > > I'm proposing: > > > > 1. Make the inheritance implementation of headers consistent with what we > > had with other record context fields. I.e. pass through the record > context > > in `context.forward()`. Note that within a processor node, users can > > already manipulate the Headers with the given APIs, so at the time of > > forwarding, the library can just copy what-ever is left / updated to the > > next processor node. > > > > 2. In the sink node, where a record is being sent to the Kafka topic, we > > should consider the following: > > > > a. For sink topics, we will set the headers into the producer record. > > b. For repartition topics, we will the headers into the producer record. > > c. For changelog topics, we will drop the headers in the produce record > > since they will not be used in restoration and not stored in the state > > store either. > > > > > > We can discuss about extending the current protocol and how to enable > users > > override those rule, and how to expose them in the DSL layer in a future > > KIP. > > > > > > > > Guozhang > > > > > > On Mon, May 7, 2018 at 5:49 PM, Matthias J. Sax <matth...@confluent.io> > > wrote: > > > >> Guozhang, > >> > >> if you advocate to forward headers by default, it might be a better > >> default strategy do forward the headers for all operators (similar to > >> topic/partition/offset metadata). It's usually harder for users to > >> reason about different cases and thus I would prefer to have consistent > >> behavior, ie, only one default strategy instead of introducing different > >> cases. > >> > >> Btw: My argument about dropping headers by default only implies, that > >> users need to copy the headers explicitly to the output records in there > >> code of they want to inspect them later -- it does not imply that > >> headers cannot be forwarded downstream. (Not sure if this was clear). > >> > >> I am also ok with copying be default thought (for me, it's a 51/49 > >> preference for dropping by default only). > >> > >> > >> -Matthias > >> > >> On 5/7/18 4:52 PM, Guozhang Wang wrote: > >>> Hi Matthias, > >>> > >>> My concern of setting `null` in all cases is that it would make headers > >> not > >>> very useful in KIP-244 then, because headers will only be available at > >> the > >>> source stream / table, but not in any of the following instances. In > >>> practice users may be more likely to look into the headers later in the > >>> pipeline. Personally I'd suggest we pass the headers for all stateless > >>> operators in DSL and everywhere in PAPI's context.forward(). For > >>> repartition topics and sink topics, we also set them in the produced > >>> records accordingly; for changelog topics, we do not set them since > they > >>> are not going to be used anywhere in the store. > >>> > >>> > >>> Guozhang > >>> > >>> > >>> On Sun, May 6, 2018 at 9:03 PM, Matthias J. Sax <matth...@confluent.io > > > >>> wrote: > >>> > >>>> I agree, that we should not block this KIP if possible. Nevertheless, > we > >>>> should try to get a reasonable default strategy for inheriting the > >>>> headers so we don't need to change it later on. > >>>> > >>>> Let's see what other think. I still tend slightly to set to `null` by > >>>> default for all cases. If the default strategy is different for > >>>> different operators as you suggest, it might be confusion to users. > >>>> IMHO, the default behavior should be as simple as possible. > >>>> > >>>> > >>>> -Matthias > >>>> > >>>> > >>>> On 5/6/18 8:53 PM, Guozhang Wang wrote: > >>>>> Matthias, thanks for sharing your opinions in the inheritance > protocol > >> of > >>>>> the record context. I'm thinking maybe we should make this discussion > >> as > >>>> a > >>>>> separate KIP by itself? If yes, then KIP-244's scope would be > smaller, > >>>> and > >>>>> within KIP-244 we can have a simple inheritance rule that setting it > to > >>>>> null when 1) going through stateful operators and 2) sending to any > >>>> topics. > >>>>> > >>>>> > >>>>> Guozhang > >>>>> > >>>>> On Sun, May 6, 2018 at 10:24 AM, Matthias J. Sax < > >> matth...@confluent.io> > >>>>> wrote: > >>>>> > >>>>>> Making the inheritance protocol a public contract seems reasonable > to > >>>> me. > >>>>>> > >>>>>> In the current implementation, all output records inherits the > offset, > >>>>>> timestamp, topic, and partition metadata from the input record. We > >>>>>> already added an API to change the timestamp explicitly for the > output > >>>>>> record thought. > >>>>>> > >>>>>> I think it make sense to keep the inheritance of offset, topic, and > >>>>>> partition. For headers, it's worth to discuss. I see arguments for > two > >>>>>> strategies: (1) inherit by default, (2) set `null` by default. > >>>>>> Independent of the default behavior, we should add an API to set > >> headers > >>>>>> for output records explicitly though (similar to the "set timestamp > >>>> API"). > >>>>>> > >>>>>> From my point of view, timestamp/headers are a different > >>>>>> "class/category" of data/metadata than topic/partition/offset. For > the > >>>>>> first category, it makes sense to manipulate them and it's more than > >>>>>> "plain metadata"; especially the timestamp. For the second category > it > >>>>>> does not make sense to manipulate it, and to me > topic/partition/offset > >>>>>> is pure metadata only---strictly speaking, it's even questionable if > >>>>>> output records should have any value for topic/partition/offset in > the > >>>>>> first place, or if they should be `null`, because those attributes > do > >>>>>> only make sense for source records that are consumed from a topic > >>>>>> directly only. On the other hand, if we make this difference > explicit, > >>>>>> it might be useful information for the use to track the current > >>>>>> topic/partition/offset of the original source record. > >>>>>> > >>>>>> Furthermore, to me, timestamps and headers are somewhat different, > >> too. > >>>>>> For stream processing it's required that every record has a > timestamp; > >>>>>> thus, it make sense to inherit the input record timestamp by default > >> (a > >>>>>> timestamp is not really metadata but actually equally important to > key > >>>>>> and value from my point of view). Header however are optional, and > >> thus > >>>>>> inheriting them is not really required. It might be convenient > though: > >>>>>> for example, imagine a simple "filter-only" application -- it would > be > >>>>>> cumbersome for users to explicitly copy the headers from the input > >>>>>> records to the output records -- it seems to be unnecessary > >> boilerplate > >>>>>> code. On the other hand, for any other more complex use case, it's > >>>>>> questionable to inherit headers---note, that headers would be > written > >> to > >>>>>> the output topics increasing the size of the messages. Overall, I am > >> not > >>>>>> sure which default strategy might be the better one for headers. Is > >>>>>> there a convincing argument for either one of them? I slightly tend > to > >>>>>> think that using `null` as default might be better. > >>>>>> > >>>>>> Last, we could also make the default behavior configurable. > Something > >>>>>> like `inherit.record.headers=true/false` with default value "false". > >>>>>> This would allow people to opt-in for auto-header-inheritance. Just > an > >>>>>> idea I wanted to add to the discussion---not sure if it's a good > one. > >>>>>> > >>>>>> > >>>>>> -Matthias > >>>>>> > >>>>>> On 5/4/18 3:13 PM, Guozhang Wang wrote: > >>>>>>> Hello Jorge, > >>>>>>> > >>>>>>>> Agree. Probably point 3 handles this. `Headers` been part of > >>>>>> `RecordContext` > >>>>>>> would be handled the same way as other attributes. > >>>>>>> > >>>>>>> Today we do not have a clear inheritance protocol for other fields > of > >>>>>>> RecordContext yet: although internally we do have some criterion on > >>>>>>> topic/partition/offset and timestamp, they are not explicitly > exposed > >>>> to > >>>>>>> users. > >>>>>>> > >>>>>>> > >>>>>>> I think we still need to have a defined protocol for headers > itself, > >>>> but > >>>>>> I > >>>>>>> agree that it better to be scoped out side of this KIP, since this > >>>>>>> inheritance protocol itself for all the fields of RecordContext > would > >>>>>>> better be a separate KIP. We can document this clearly in the wiki > >>>> page. > >>>>>>> > >>>>>>> Guozhang > >>>>>>> > >>>>>>> > >>>>>>> On Fri, May 4, 2018 at 5:26 AM, Florian Garcia < > >>>>>>> garcia.florian.pe...@gmail.com> wrote: > >>>>>>> > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> For me this is a great first step to have Headers in streaming. > >>>>>>>> My current use case is about distributed tracing (Zipkin) and with > >> the > >>>>>>>> headers in the processorContext() I'll be able to manage that for > >> the > >>>>>> most > >>>>>>>> cases. > >>>>>>>> The KIP-159 should follow after this but this is where all the > major > >>>>>>>> questions will arise for stateful operations (as Guozhang said). > >>>>>>>> > >>>>>>>> Thanks for the work on this Jorge. > >>>>>>>> > >>>>>>>> Le ven. 4 mai 2018 à 01:04, Jorge Esteban Quilcate Otoya < > >>>>>>>> quilcate.jo...@gmail.com> a écrit : > >>>>>>>> > >>>>>>>>> Thanks Guozhang and John for your feedback. > >>>>>>>>> > >>>>>>>>>> 1. We need to have a clear inheritance protocol of headers in > our > >>>>>>>>> topology: > >>>>>>>>>> 1.a. In PAPI's context.forward() call, it should be > >>>> straight-forward. > >>>>>>>>>> 1.b. In DSL stateless operators, it should be straight-forward. > >>>>>>>>>> 1.c. What about in stateful operators like aggregates and joins? > >>>>>>>>> > >>>>>>>>> Agree. Probably point 3 handles this. `Headers` been part of > >>>>>>>>> `RecordContext` would be handled the same way as other > attributes. > >>>>>>>>> > >>>>>>>>>> 3. In future work "Adding DSL Processors to use Headers to > >>>>>>>>> filter/map/branch", > >>>>>>>>> it may well be covered in KIP-159; worth taking a look at that > KIP. > >>>>>>>>> > >>>>>>>>> Yes, I will point to it. > >>>>>>>>> > >>>>>>>>>> 2. In terms of internal implementations, should the state store > >>>>>>>>> cache include the headers then in order to be sent downstreams? > >>>>>>>>> > >>>>>>>>> Good question. As `LRUCacheEntry` extends `RecordContext`, I > thinks > >>>>>> this > >>>>>>>> is > >>>>>>>>> already supported. I will detail this on the KIP. > >>>>>>>>> > >>>>>>>>>> 4. MINOR: "void process(K key, V value, Headers headers)", this > >>>> should > >>>>>>>> be > >>>>>>>>> removed? > >>>>>>>>> > >>>>>>>>> Fixed, thanks. > >>>>>>>>> > >>>>>>>>>> 5. MINOR: it seems to be the case that in this KIP, our scope is > >>>> only > >>>>>>>>> for exposing > >>>>>>>>> the headers for reading, and not allowing users to add / modify > >>>>>> headers, > >>>>>>>>> right? If yes, we'd better state it clearly at the "Proposed > >> Changes" > >>>>>>>>> section. > >>>>>>>>> > >>>>>>>>> As headers is exposed in the `ProcessContext`, and headers will > be > >>>> send > >>>>>>>>> downstream, it can be mutated (add/remove headers). > >>>>>>>>> > >>>>>>>>> > Also, despite the decreased scope in this KIP, I think it > might > >> be > >>>>>>>>> valuable to define what will happen to headers once this change > is > >>>>>>>>> implemented. For example, I think a minimal groundwork-level > change > >>>>>> might > >>>>>>>>> be to make the API changes, while promising to drop all headers > >> from > >>>>>>>> input > >>>>>>>>> records. > >>>>>>>>> > >>>>>>>>> I will suggest to pass headers to downstream nodes, and don't > drop > >>>>>> yhrm. > >>>>>>>>> Clients will have to drop `Headers` if they have used them. > >>>>>>>>> Or it could be something like a boolean config property that > manage > >>>>>> this. > >>>>>>>>> I would like to hear feedback here. > >>>>>>>>> > >>>>>>>>>> A maximal groundwork change would be to forward the headers > >> through > >>>>>> all > >>>>>>>>> operators > >>>>>>>>> in > >>>>>>>>> > >>>>>>>>> Streams. But I think there are some unresolved questions about > >>>>>>>> forwarding, > >>>>>>>>> like "what happens to the headers in a join?" > >>>>>>>>> Probably this would be solve once KIP-159 is implemented and > >>>> supporting > >>>>>>>>> Headers. > >>>>>>>>> > >>>>>>>>>> There's of course some middle ground, but instinctively, I think > >> I'd > >>>>>>>>> prefer to have a clear definition that headers are currently > *not* > >>>>>>>>> forwarded, rather than having a complex list of operators that do > >> or > >>>>>>>> don't > >>>>>>>>> forward them. Plus, I think it might be tricky to define this > >>>> behavior > >>>>>>>>> while not allowing the scope to return to that of your original > >>>>>> proposal! > >>>>>>>>> > >>>>>>>>> Agree. But `Headers` were forwarded *explicitly* in the original > >>>>>>>> proposal. > >>>>>>>>> The current one pass it as part of `RecordContext`, so if it's > >>>> forward > >>>>>> it > >>>>>>>>> or not is as the same as `RecordContext`. > >>>>>>>>> On top of this implementation, we can design how filter/map/join > >> will > >>>>>> be > >>>>>>>>> handled. Probably following KIP-159 approach. > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Jorge. > >>>>>>>>> > >>>>>>>>> El mié., 2 may. 2018 a las 22:56, Guozhang Wang (< > >> wangg...@gmail.com > >>>>> ) > >>>>>>>>> escribió: > >>>>>>>>> > >>>>>>>>>> Hi Jorge, > >>>>>>>>>> > >>>>>>>>>> Thanks for the written KIP! Made a pass over it and left some > >>>> comments > >>>>>>>>>> (some of them overlapped with John's): > >>>>>>>>>> > >>>>>>>>>> 1. We need to have a clear inheritance protocol of headers in > our > >>>>>>>>> topology: > >>>>>>>>>> > >>>>>>>>>> 1.a. In PAPI's context.forward() call, it should be > >>>> straight-forward. > >>>>>>>>>> 1.b. In DSL stateless operators, it should be straight-forward. > >>>>>>>>>> 1.c. What about in stateful operators like aggregates and joins? > >>>>>>>>>> > >>>>>>>>>> 2. In terms of internal implementations, should the state store > >>>> cache > >>>>>>>>>> include the headers then in order to be sent downstreams? > >>>>>>>>>> > >>>>>>>>>> 3. In future work "Adding DSL Processors to use Headers to > >>>>>>>>>> filter/map/branch", it may well be covered in KIP-159; worth > >> taking > >>>> a > >>>>>>>>> look > >>>>>>>>>> at that KIP. > >>>>>>>>>> > >>>>>>>>>> 4. MINOR: "void process(K key, V value, Headers headers)", this > >>>> should > >>>>>>>> be > >>>>>>>>>> removed? > >>>>>>>>>> > >>>>>>>>>> 5. MINOR: it seems to be the case that in this KIP, our scope is > >>>> only > >>>>>>>> for > >>>>>>>>>> exposing the headers for reading, and not allowing users to add > / > >>>>>>>> modify > >>>>>>>>>> headers, right? If yes, we'd better state it clearly at the > >>>> "Proposed > >>>>>>>>>> Changes" section. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Guozhang > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Wed, May 2, 2018 at 8:42 AM, John Roesler <j...@confluent.io > > > >>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Jorge, > >>>>>>>>>>> > >>>>>>>>>>> Thanks for the design work. > >>>>>>>>>>> > >>>>>>>>>>> I agree that de-scoping the work to just the Processor API will > >>>> help > >>>>>>>>>>> contain the design and implementation complexity. > >>>>>>>>>>> > >>>>>>>>>>> In the KIP, it mentions that the headers would be available in > >> the > >>>>>>>>>>> ProcessorContext, (like "context.headers()"). It also says that > >>>>>>>>>>> implementers would need to implement the method "void process(K > >>>> key, > >>>>>>>> V > >>>>>>>>>>> value, Headers headers);". I think maybe you meant to remove > the > >>>>>>>>> proposal > >>>>>>>>>>> to modify "process", since it wouldn't be necessary in > >> conjunction > >>>>>>>> with > >>>>>>>>>> the > >>>>>>>>>>> ProcessorContext change, and it's not represented in your PR. > >>>>>>>>>>> > >>>>>>>>>>> Also, despite the decreased scope in this KIP, I think it might > >> be > >>>>>>>>>> valuable > >>>>>>>>>>> to define what will happen to headers once this change is > >>>>>>>> implemented. > >>>>>>>>>> For > >>>>>>>>>>> example, I think a minimal groundwork-level change might be to > >> make > >>>>>>>> the > >>>>>>>>>> API > >>>>>>>>>>> changes, while promising to drop all headers from input > records. > >>>>>>>>>>> > >>>>>>>>>>> A maximal groundwork change would be to forward the headers > >> through > >>>>>>>> all > >>>>>>>>>>> operators in Streams. But I think there are some unresolved > >>>> questions > >>>>>>>>>> about > >>>>>>>>>>> forwarding, like "what happens to the headers in a join?" > >>>>>>>>>>> > >>>>>>>>>>> There's of course some middle ground, but instinctively, I > think > >>>> I'd > >>>>>>>>>> prefer > >>>>>>>>>>> to have a clear definition that headers are currently *not* > >>>>>>>> forwarded, > >>>>>>>>>>> rather than having a complex list of operators that do or don't > >>>>>>>> forward > >>>>>>>>>>> them. Plus, I think it might be tricky to define this behavior > >>>> while > >>>>>>>>> not > >>>>>>>>>>> allowing the scope to return to that of your original proposal! > >>>>>>>>>>> > >>>>>>>>>>> Thanks again for the KIP, > >>>>>>>>>>> -John > >>>>>>>>>>> > >>>>>>>>>>> On Wed, May 2, 2018 at 8:05 AM, Jorge Esteban Quilcate Otoya < > >>>>>>>>>>> quilcate.jo...@gmail.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Matthias, > >>>>>>>>>>>> > >>>>>>>>>>>> I've created a new JIRA to track this, updated the KIP and > >> create > >>>> a > >>>>>>>>> PR. > >>>>>>>>>>>> > >>>>>>>>>>>> Looking forward to your feedback, > >>>>>>>>>>>> > >>>>>>>>>>>> Jorge. > >>>>>>>>>>>> > >>>>>>>>>>>> El mar., 13 feb. 2018 a las 22:43, Matthias J. Sax (< > >>>>>>>>>>> matth...@confluent.io > >>>>>>>>>>>>> ) > >>>>>>>>>>>> escribió: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi Jorge, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I would like to unblock this KIP to make some progress. The > >>>>>>>> tricky > >>>>>>>>>>>>> question of this work, seems to be how to expose headers at > DSL > >>>>>>>>>> level. > >>>>>>>>>>>>> This related to KIP-149 and KIP-159. However, for Processor > >> API, > >>>>>>>> it > >>>>>>>>>>>>> seems to be rather straight forward to add headers to the > API. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thus, I would suggest to de-scope this KIP and add header > >> support > >>>>>>>>> for > >>>>>>>>>>>>> Processor API only as a first step. If this is done, we can > see > >>>>>>>> in > >>>>>>>>> a > >>>>>>>>>>>>> second step, how to add headers at DSL level. > >>>>>>>>>>>>> > >>>>>>>>>>>>> WDYT about this proposal? > >>>>>>>>>>>>> > >>>>>>>>>>>>> If you agree, please update the JIRA and KIP accordingly. > Note, > >>>>>>>>> that > >>>>>>>>>> we > >>>>>>>>>>>>> have two JIRA that are duplicates atm. We can scope them > >>>>>>>>> accordingly: > >>>>>>>>>>>>> one for PAPI only, and second as a dependent JIRA for DSL. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 12/30/17 3:11 PM, Jorge Esteban Quilcate Otoya wrote: > >>>>>>>>>>>>>> Thanks for your feedback! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 1. I was adding headers to KeyValue to support groupBy, but > I > >>>>>>>>> think > >>>>>>>>>>> it > >>>>>>>>>>>> is > >>>>>>>>>>>>>> not necessary. It should be enough with mapping headers to > >>>>>>>>>> key/value > >>>>>>>>>>>> and > >>>>>>>>>>>>>> then group using current KeyValue structure. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2. Yes. IMO key/value stores, like RocksDB, rely on KV as > >>>>>>>>>> structure, > >>>>>>>>>>>>> hence > >>>>>>>>>>>>>> considering headers as part of stateful operations will not > >> fit > >>>>>>>>> in > >>>>>>>>>>> this > >>>>>>>>>>>>>> approach and increase complexity (I cannot think in a > use-case > >>>>>>>>> that > >>>>>>>>>>>> need > >>>>>>>>>>>>>> this). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 3. and 4. Changes on 1. will solve this issue. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Probably I rush a bit proposing this change, I was not aware > >> of > >>>>>>>>>>> KIP-159 > >>>>>>>>>>>>> or > >>>>>>>>>>>>>> KAFKA-5632. > >>>>>>>>>>>>>> If KIP-159 is adopted and we reduce this KIP to add Headers > to > >>>>>>>>>>>>>> RecordContext will be enough, but I'm not sure about the > scope > >>>>>>>> of > >>>>>>>>>>>>> KIP-159. > >>>>>>>>>>>>>> If it includes stateful operations will be difficult to > >>>>>>>>> implemented > >>>>>>>>>>> as > >>>>>>>>>>>>>> stated in 2. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>> Jorge. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> El mar., 26 dic. 2017 a las 20:04, Matthias J. Sax (< > >>>>>>>>>>>>> matth...@confluent.io>) > >>>>>>>>>>>>>> escribió: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thanks for the KIP Jorge, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> As Bill pointed out already, we should be careful with > adding > >>>>>>>>> new > >>>>>>>>>>>>>>> overloads as this contradicts the work done via KIP-182. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This KIP also seems to be related to KIP-149 and KIP-159. > Are > >>>>>>>>> you > >>>>>>>>>>>> aware > >>>>>>>>>>>>>>> of them? Both have quite long DISCUSS threads, but it might > >> be > >>>>>>>>>> worth > >>>>>>>>>>>>>>> browsing through them. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> A few further questions: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - why do you want to add the headers to `KeyValue`? I am > not > >>>>>>>>> sure > >>>>>>>>>>> if > >>>>>>>>>>>> we > >>>>>>>>>>>>>>> should consider headers as optional metadata and add it to > >>>>>>>>>>>>>>> `RecordContext` similar to timestamp, offset, etc. only > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - You only include stateless single-record transformations > >> at > >>>>>>>>> the > >>>>>>>>>>> DSL > >>>>>>>>>>>>>>> level. Do you suggest that all other operator just drop > >>>>>>>> headers > >>>>>>>>> on > >>>>>>>>>>> the > >>>>>>>>>>>>>>> floor? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - Why do you only want to put headers into in-memory and > >>>>>>>> cache > >>>>>>>>>> but > >>>>>>>>>>>> not > >>>>>>>>>>>>>>> RocksDB store? What do you mean by "pass through"? IMHO, > all > >>>>>>>>>> stores > >>>>>>>>>>>>>>> should behave the same at DSL level. > >>>>>>>>>>>>>>> -> if we store the headers in the state stores, what is > >> the > >>>>>>>>>>> upgrade > >>>>>>>>>>>>>>> path? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> - Why do we need to store record header in state in the > >> first > >>>>>>>>>>> place, > >>>>>>>>>>>> if > >>>>>>>>>>>>>>> we exclude stateful operator at DSL level? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> What is the motivation for the "border lines" you choose? > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -Matthias > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On 12/21/17 8:18 AM, Bill Bejeck wrote: > >>>>>>>>>>>>>>>> Jorge, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for the KIP, I know this is a feature others in the > >>>>>>>>>>> community > >>>>>>>>>>>>> have > >>>>>>>>>>>>>>>> been interested in getting into Kafka Streams. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I took a quick pass over it, and I have one initial > >> question. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> We recently reduced overloads with KIP-182, and in this > KIP > >>>>>>>> we > >>>>>>>>>> are > >>>>>>>>>>>>>>>> increasing them again. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> I can see from the KIP why they are necessary, but I'm > >>>>>>>>> wondering > >>>>>>>>>> if > >>>>>>>>>>>>> there > >>>>>>>>>>>>>>>> is something else we can do to cut down on the overloads > >>>>>>>>>>>> introduced. I > >>>>>>>>>>>>>>>> don't have any sound suggestions ATM, so I'll have to > think > >>>>>>>>> about > >>>>>>>>>>> it > >>>>>>>>>>>>> some > >>>>>>>>>>>>>>>> more, but I wanted to put the thought out there. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>>>> Bill > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Thu, Dec 21, 2017 at 9:06 AM, Jorge Esteban Quilcate > >>>>>>>> Otoya < > >>>>>>>>>>>>>>>> quilcate.jo...@gmail.com> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi all, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have created a KIP to add Record Headers support to > Kafka > >>>>>>>>>>> Streams > >>>>>>>>>>>>> API: > >>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>>>>>>>>>>> 244%3A+Add+Record+Header+support+to+Kafka+Streams > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The main goal is to be able to use headers to filter, map > >>>>>>>> and > >>>>>>>>>>>> process > >>>>>>>>>>>>>>>>> records as streams. Stateful processing (joins, windows) > >> are > >>>>>>>>> not > >>>>>>>>>>>>>>>>> considered. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Proposed changes/Draft: > >>>>>>>>>>>>>>>>> https://github.com/apache/kafka/compare/trunk...jeqo: > >>>>>>>>>>>> streams-headers > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Feedback and suggestions are more than welcome. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Jorge. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> -- Guozhang > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >>> > >> > >> > > > > > >
