Re: KTable and cleanup.policy=compact

2017-02-13 Thread Matthias J. Sax
As you read the KTable from a topic via KStreamBuilder#table("my-table-topic") you should set log cleanup policy to "compacted" for "my-table-topic"). -Matthias On 2/13/17 4:49 AM, Eno Thereska wrote: > Hi Jon, > > If I understand your question correctly: > - any new KTables created by the DSL

Re: KTable and cleanup.policy=compact

2017-02-13 Thread Eno Thereska
Hi Jon, If I understand your question correctly: - any new KTables created by the DSL will automatically get the right policy. You don't need to do anything special. - otherwise you'll have to set the policy on the Kafka topic. Eno > On 13 Feb 2017, at 11:16, Jon Yeargers

Re: KTable and cleanup.policy=compact

2017-02-13 Thread Jon Yeargers
If Im doing a KStream.leftJoin(KTable) how would I set this configuration for just the KTable portion? IE I have KStream = KStreamBuilder.stream() KTable = KStreamBuilder.table() ... (join occurs.. data flows.. ppl are brought closer together.. there is peace in the valley.. for me... ) ...

Re: KTable and cleanup.policy=compact

2017-02-08 Thread Eno Thereska
Yeah makes sense. I was looking at it from the point of view of keeping all data forever. Eno > On 8 Feb 2017, at 20:27, Matthias J. Sax wrote: > > Yes, that could happen if a key was not updated for a longer period than > topic retention time. > > If you want to force

Re: KTable and cleanup.policy=compact

2017-02-08 Thread Matthias J. Sax
Yes, that could happen if a key was not updated for a longer period than topic retention time. If you want to force a changelog creation, you can do a dummy aggregate instead of using KStreamBuilder#table() > KTable table = KStreamBuilder.stream("topic").groupByKey().reduce(new > Reducer() { >

Re: KTable and cleanup.policy=compact

2017-02-08 Thread Mathieu Fenniak
I think there could be correctness implications... the default cleanup.policy of delete would mean that topic entries past the retention policy might have been removed. If you scale up the application, new application instances won't be able to restore a complete table into its local state store.

Re: KTable and cleanup.policy=compact

2017-02-08 Thread Eno Thereska
If you fail to set the policy to compact, there shouldn't be any correctness implications, however your topics will grow larger than necessary. Eno > On 8 Feb 2017, at 18:56, Jon Yeargers wrote: > > What are the ramifications of failing to do this? > > On Tue, Feb

Re: KTable and cleanup.policy=compact

2017-02-08 Thread Jon Yeargers
What are the ramifications of failing to do this? On Tue, Feb 7, 2017 at 9:16 PM, Matthias J. Sax wrote: > Yes, that is correct. > > > -Matthias > > > On 2/7/17 6:39 PM, Mathieu Fenniak wrote: > > Hey kafka users, > > > > Is it correct that a Kafka topic that is used for

Re: KTable and cleanup.policy=compact

2017-02-07 Thread Matthias J. Sax
Yes, that is correct. -Matthias On 2/7/17 6:39 PM, Mathieu Fenniak wrote: > Hey kafka users, > > Is it correct that a Kafka topic that is used for a KTable should be set to > cleanup.policy=compact? > > I've never noticed until today that the KStreamBuilder#table() > documentation says:

KTable and cleanup.policy=compact

2017-02-07 Thread Mathieu Fenniak
Hey kafka users, Is it correct that a Kafka topic that is used for a KTable should be set to cleanup.policy=compact? I've never noticed until today that the KStreamBuilder#table() documentation says: "However, no internal changelog topic is created since the original input topic can be used for