Yes you are correct Elliot. If you look at it just from the point of view of
what's in the topic, and how the topic is configured (e.g., with "compact")
there is no difference.
Thanks
Eno
> On 11 Feb 2017, at 17:56, Elliot Crosby-McCullough <elliot...@gmail.com>
> wrote:
>
> For my own clarity, is there any actual distinction between
> `stream.to('topic')`
> where `topic` is set to compact and the upcoming `stream.toTable('topic')`
> if you're not going to immediately use the table in this topology, i.e. if
> you want to use it as a table in some other processor application? Am I
> right in my understanding that the table aspect is related to behaviour
> within this specific processor (GlobalKTables aside)?
>
> On 10 February 2017 at 21:59, Matthias J. Sax <matth...@confluent.io> wrote:
>
>> I agree that the API can be improved and we are working on that.
>>
>> Btw: KStream#toTable() was already suggested in KIP-114 discussion:
>>
>> http://search-hadoop.com/m/Kafka/uyzND19QaLMqiR2e1?subj=
>> Re+DISCUSS+KIP+114+KTable+materialization+and+improved+semantics
>>
>>
>> However for now, you can only choose from the two options as described.
>>
>>
>> -Matthias
>>
>>
>>
>> On 2/10/17 1:49 PM, Nick DeCoursin wrote:
>>> To be honest, I don't think either of these options are very good.
>>>
>>>
>>> stream.to("some-other-topic");
>>> builder.table("some-other-topic");
>>>
>>>
>>> As explained here
>>> <http://mail-archives.apache.org/mod_mbox/kafka-users/
>> 201702.mbox/browser>,
>>> if the underlying topic doesn't have cleanup.policy=compact, the state
>>> store may invisibly lose data!
>>>
>>> As for the second option, it's obviously verbose, and it doesn't let me
>>> choose the topic name. As a result, I end with some long weird changelog
>>> topic name; and, some of the Kafka Connectors map topics to the table
>> name,
>>> eeeh.
>>>
>>> To improve this, just add a topic parameter to both of the overloaded
>>> KGroupedStream.reduce methods:
>>>
>>> reduce(Reducer<V> reducer, String, topicName, String storeName)
>>>
>>>
>>> Furthermore, there should be a short cut to the second option, like this:
>>>
>>> // Creates a KTable from the KStream where
>>> // the key of the KStream is the key of the KTable.
>>> // Any latter key overwrites the former.
>>> someStream.table(
>>> Serde<K>,
>>> Serde<V>,
>>> topicName,
>>> tableName
>>> );
>>>
>>>
>>> or maybe the Serdes can be inferred? Either way, this would be a nice
>> clean
>>> approach to a (maybe?) common use case.
>>>
>>> Thank you,
>>> Nick
>>>
>>> On 25 January 2017 at 07:46, Nick DeCoursin <n.decour...@foodpanda.com>
>>> wrote:
>>>
>>>> Thank you very much, both suggestions are wonderful, and I will try
>> them.
>>>> Have a great day!
>>>>
>>>> Kind regards,
>>>> Nick
>>>>
>>>> On 24 January 2017 at 19:46, Matthias J. Sax <matth...@confluent.io>
>>>> wrote:
>>>>
>>>>> If your data is already partitioned by key, you can save writing to a
>>>>> topic by doing a dummy reduce instead:
>>>>>
>>>>> stream
>>>>> .groupByKey()
>>>>> .reduce(new Reducer() {
>>>>> V apply(V value1, V value2) {
>>>>> return value2;
>>>>> }
>>>>> },
>>>>> "yourStoreName");
>>>>>
>>>>> (replace V with your actuall value type)
>>>>>
>>>>>
>>>>> -Matthias
>>>>>
>>>>> On 1/24/17 8:53 AM, Damian Guy wrote:
>>>>>> Hi Nick,
>>>>>>
>>>>>> I guess there is some reason why you can't just build it as a table to
>>>>>> begin with?
>>>>>> There isn't a convenient method for doing this right now, but you
>> could
>>>>> do
>>>>>> something like:
>>>>>>
>>>>>> stream.to("some-other-topic");
>>>>>> builder.table("some-other-topic");
>>>>>>
>>>>>> Thanks,
>>>>>> Damian
>>>>>>
>>>>>> On Tue, 24 Jan 2017 at 16:32 Nick DeCoursin <
>> n.decour...@foodpanda.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> How can I simply table a Kafka Stream? I have a Kafka Stream, and I
>>>>> want to
>>>>>>> create a table from it backed by a state store. The key of the stream
>>>>> could
>>>>>>> be the same as the table.
>>>>>>>
>>>>>>> I've tried following examples, but it seems all examples use
>> `groupBy`
>>>>> or
>>>>>>> `count` to convert `KStream`s into `KTable`s. Is there any other way?
>>>>>>>
>>>>>>> Thank you very much,
>>>>>>> Nick DeCoursin
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Nick DeCoursin
>>>>>>> Software Engineer
>>>>>>> foodpanda
>>>>>>>
>>>>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>>>>>>
>>>>>>> Mail | n.decour...@foodpanda.com
>>>>>>>
>>>>>>> Skype | nick.foodpanda
>>>>>>>
>>>>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>>>>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>>>>>>> USt-ID-Nr | DE 283789080
>>>>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>>>>>>
>>>>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>>>>>>> confidential and may be privileged. It may be read, copied and used
>>>>> only by
>>>>>>> the intended recipient. If you have received it in error please
>>>>> contact the
>>>>>>> sender (by return e-mail) immediately and delete this message. Any
>>>>>>> unauthorized use or dissemination of this message in whole or in
>> parts
>>>>> is
>>>>>>> strictly prohibited.
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Nick DeCoursin
>>>> Software Engineer
>>>> foodpanda
>>>>
>>>> Tel | +1 920 450 5434 <(920)%20450-5434>
>>>>
>>>> Mail | n.decour...@foodpanda.com
>>>>
>>>> Skype | nick.foodpanda
>>>>
>>>> Foodpanda GmbH | Schreiberhauer Str. 30 | 10317 Berlin | Germany
>>>> Sitz der Gesellschaft | Berlin, AG Charlottenburg | HRB 138224 B |
>>>> USt-ID-Nr | DE 283789080
>>>> Geschäftsführer | Benjamin Bauer, Felix Plog, Ralf Wenzel
>>>>
>>>> CONFIDENTIALITY NOTICE: This message (including any attachments) is
>>>> confidential and may be privileged. It may be read, copied and used
>> only by
>>>> the intended recipient. If you have received it in error please contact
>> the
>>>> sender (by return e-mail) immediately and delete this message. Any
>>>> unauthorized use or dissemination of this message in whole or in parts
>> is
>>>> strictly prohibited.
>>>>
>>>
>>>
>>>
>>
>>
