Table a KStream

Hello, How can I simply table a Kafka Stream? I have a Kafka Stream, and I want to create a table from it backed by a state store. The key of the stream could be the same as the table. I've tried following examples, but it seems all examples use `groupBy` or `count` to convert `KStream`s into `KT

Re: Table a KStream

Hi Nick, I guess there is some reason why you can't just build it as a table to begin with? There isn't a convenient method for doing this right now, but you could do something like: stream.to("some-other-topic"); builder.table("some-other-topic"); Thanks, Damian On Tue, 24 Jan 2017 at 16:32 Ni

Re: Table a KStream

If your data is already partitioned by key, you can save writing to a topic by doing a dummy reduce instead: stream .groupByKey() .reduce(new Reducer() { V apply(V value1, V value2) { return value2; } }, "yourStoreName"); (replace V with your actuall value type) -Matthias

Re: Table a KStream

Thank you very much, both suggestions are wonderful, and I will try them. Have a great day! Kind regards, Nick On 24 January 2017 at 19:46, Matthias J. Sax wrote: > If your data is already partitioned by key, you can save writing to a > topic by doing a dummy reduce instead: > > stream > .gro

Re: Table a KStream

To be honest, I don't think either of these options are very good. stream.to("some-other-topic"); builder.table("some-other-topic"); As explained here , if the underlying topic doesn't have cleanup.policy=compact, the sta

Re: Table a KStream

I agree that the API can be improved and we are working on that. Btw: KStream#toTable() was already suggested in KIP-114 discussion: http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics However for now, you can only choose from

Re: Table a KStream

Sounds good, thank you! Kind regards, Nick On 10 February 2017 at 22:59, Matthias J. Sax wrote: > I agree that the API can be improved and we are working on that. > > Btw: KStream#toTable() was already suggested in KIP-114 discussion: > > http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=

Re: Table a KStream

For my own clarity, is there any actual distinction between `stream.to('topic')` where `topic` is set to compact and the upcoming `stream.toTable('topic')` if you're not going to immediately use the table in this topology, i.e. if you want to use it as a table in some other processor application?

Re: Table a KStream

Yes you are correct Elliot. If you look at it just from the point of view of what's in the topic, and how the topic is configured (e.g., with "compact") there is no difference. Thanks Eno > On 11 Feb 2017, at 17:56, Elliot Crosby-McCullough > wrote: > > For my own clarity, is there any act

Re: Table a KStream

Thanks! On 11 February 2017 at 18:39, Eno Thereska wrote: > Yes you are correct Elliot. If you look at it just from the point of view > of what's in the topic, and how the topic is configured (e.g., with > "compact") there is no difference. > > > Thanks > Eno > > > On 11 Feb 2017, at 17:56, Elli