Thanks!
On 11 February 2017 at 18:39, Eno Thereska <eno.there...@gmail.com> wrote:
> Yes you are correct Elliot. If you look at it just from the point of view
> of what's in the topic, and how the topic is configured (e.g., with
> "compact") there is no difference.
>
>
> Thanks
> Eno
>
> > On 11 Feb 2017, at 17:56, Elliot Crosby-McCullough <elliot...@gmail.com>
> wrote:
> >
> > For my own clarity, is there any actual distinction between
> > `stream.to('topic')`
> > where `topic` is set to compact and the upcoming
> `stream.toTable('topic')`
> > if you're not going to immediately use the table in this topology, i.e.
> if
> > you want to use it as a table in some other processor application? Am I
> > right in my understanding that the table aspect is related to behaviour
> > within this specific processor (GlobalKTables aside)?
> >
> > On 10 February 2017 at 21:59, Matthias J. Sax <matth...@confluent.io>
> wrote:
> >
> >> I agree that the API can be improved and we are working on that.
> >>
> >> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
> >>
> >> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
> >> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
> >>
> >>
> >> However for now, you can only choose from the two options as described.
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
> >>> To be honest, I don't think either of these options are very good.
> >>>
> >>>
> >>> stream.to("some-other-topic");
> >>> builder.table("some-other-topic");
> >>>
> >>>
> >>> As explained here
> >>> <http://mail-archives.apache.org/mod_mbox/kafka-users/
> >> 201702.mbox/browser>,
> >>> if the underlying topic doesn't have cleanup.policy=compact, the state
> >>> store may invisibly lose data!
> >>>
> >>> As for the second option, it's obviously verbose, and it doesn't let me
> >>> choose the topic name. As a result, I end with some long weird
> changelog
> >>> topic name; and, some of the Kafka Connectors map topics to the table
> >> name,
> >>> eeeh.
> >>>
> >>> To improve this, just add a topic parameter to both of the overloaded
> >>> KGroupedStream.reduce methods:
> >>>
> >>> reduce(Reducer<V> reducer, String, topicName, String storeName)
> >>>
> >>>
> >>> Furthermore, there should be a short cut to the second option, like
> this:
> >>>
> >>> // Creates a KTable from the KStream where
> >>> // the key of the KStream is the key of the KTable.
> >>> // Any latter key overwrites the former.
> >>> someStream.table(
> >>> Serde<K>,
> >>> Serde<V>,
> >>> topicName,
> >>> tableName
> >>> );
> >>>
> >>>
> >>> or maybe the Serdes can be inferred? Either way, this would be a nice
> >> clean
> >>> approach to a (maybe?) common use case.
> >>>
> >>> Thank you,
> >>> Nick
> >>>
> >>> On 25 January 2017 at 07:46, Nick DeCoursin <n.decour...@foodpanda.com
> >
> >>> wrote:
> >>>
> >>>> Thank you very much, both suggestions are wonderful, and I will try
> >> them.
> >>>> Have a great day!
> >>>>
> >>>> Kind regards,
> >>>> Nick
> >>>>
> >>>> On 24 January 2017 at 19:46, Matthias J. Sax <matth...@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> If your data is already partitioned by key, you can save writing to a
> >>>>> topic by doing a dummy reduce instead:
> >>>>>
> >>>>> stream
> >>>>> .groupByKey()
> >>>>> .reduce(new Reducer() {
> >>>>> V apply(V value1, V value2) {
> >>>>> return value2;
> >>>>> }
> >>>>> },
> >>>>> "yourStoreName");
> >>>>>
> >>>>> (replace V with your actuall value type)
> >>>>>
> >>>>>
> >>>>> -Matthias
> >>>>>
> >>>>> On 1/24/17 8:53 AM, Damian Guy wrote:
> >>>>>> Hi Nick,
> >>>>>>
> >>>>>> I guess there is some reason why you can't just build it as a table
> to
> >>>>>> begin with?
> >>>>>> There isn't a convenient method for doing this right now, but you
> >> could
> >>>>> do
> >>>>>> something like:
> >>>>>>
> >>>>>> stream.to("some-other-topic");
> >>>>>> builder.table("some-other-topic");
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Damian
> >>>>>>
> >>>>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
> >> n.decour...@foodpanda.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
> >>>>> want to
> >>>>>>> create a table from it backed by a state store. The key of the
> stream
> >>>>> could
> >>>>>>> be the same as the table.
> >>>>>>>
> >>>>>>> I've tried following examples, but it seems all examples use
> >> `groupBy`
> >>>>> or
> >>>>>>> `count` to convert `KStream`s into `KTable`s. Is there any other
> way?
> >>>>>>>
> >>>>>>> Thank you very much,
> >>>>>>> Nick DeCoursin
> >>>>>>>
> >>>>>>> --
> >>>>>>>
> >>>>>>> Nick DeCoursin
> >>>>>>> Software Engineer
> >>>>>>> foodpanda
> >>>>>>>
> >>>>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>>>>
> >>>>>>> Mail | n.decour...@foodpanda.com
> >>>>>>>
> >>>>>>> Skype | nick.foodpanda
> >>>>>>>
> >>>>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>>>>> USt-ID-Nr | DE 283789080
> >>>>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>>>>
> >>>>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>>>>> confidential and may be privileged. It may be read, copied and used
> >>>>> only by
> >>>>>>> the intended recipient. If you have received it in error please
> >>>>> contact the
> >>>>>>> sender (by return e-mail) immediately and delete this message. Any
> >>>>>>> unauthorized use or dissemination of this message in whole or in
> >> parts
> >>>>> is
> >>>>>>> strictly prohibited.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>>
> >>>> Nick DeCoursin
> >>>> Software Engineer
> >>>> foodpanda
> >>>>
> >>>> Tel | +1 920 450 5434 <(920)%20450-5434>
> >>>>
> >>>> Mail | n.decour...@foodpanda.com
> >>>>
> >>>> Skype | nick.foodpanda
> >>>>
> >>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
> >>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
> >>>> USt-ID-Nr | DE 283789080
> >>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
> >>>>
> >>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
> >>>> confidential and may be privileged. It may be read, copied and used
> >> only by
> >>>> the intended recipient. If you have received it in error please
> contact
> >> the
> >>>> sender (by return e-mail) immediately and delete this message. Any
> >>>> unauthorized use or dissemination of this message in whole or in parts
> >> is
> >>>> strictly prohibited.
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
>
>
