Re: KIP-675: Convert KTable to a KStream using the previous value

2020-10-09 Thread Javier Freire Riobo
I see. It's great hidden functionality and all you have to do is define a good API and expose it. I think the easiest thing would be to expose a first operator, and then add more operators if they are claimed. I think it is hidden for not complicating the API. Thank you El vie., 9 oct. 2020 a las

Re: KIP-675: Convert KTable to a KStream using the previous value

2020-10-08 Thread Matthias J. Sax
I agree that there are cases when it is useful to get the old and new value. In fact, the DSL internally often tracks old and new value via a `Change` value type. We did have some discussion that it might be useful to expose this currently internal `Change` type in the public API. But if we do thi

Re: KIP-675: Convert KTable to a KStream using the previous value

2020-10-08 Thread Javier Freire Riobo
You're right. The behavior is correct with the cache disabled. Anyway I think the operator I propose can be useful. The need to generate a value from the previous and current value of a record can be quite common. I think the only way to implement it is through an aggregate using a helper class. I

Re: KIP-675: Convert KTable to a KStream using the previous value

2020-10-08 Thread Matthias J. Sax
I guess I understand now. However, it seems to be an "issue" with record caching. Setting the commit interval to zero would flush the cache each time, but it is not the "right" config change. You should just disable the `KTable` cache instead. You can disable caching globally by setting `cache.ma

Re: KIP-675: Convert KTable to a KStream using the previous value

2020-10-07 Thread Javier Freire Riobo
I have done a small demo example. I hope it serves as a clarification. https://github.com/javierfreire/KTableToKStreamTest Thank you very much El mié., 7 oct. 2020 a las 3:01, Matthias J. Sax () escribió: > Thanks for the KIP. > > I am not sure if I understand the motivation. In particular the

Re: KIP-675: Convert KTable to a KStream using the previous value

2020-10-06 Thread Matthias J. Sax
Thanks for the KIP. I am not sure if I understand the motivation. In particular the KIP says: > The main problem, apart from needing more code, is that if the same event is > received twice at the same time and the commit time is not 0, the difference > is deleted and nothing is emitted. Can y

KIP-675: Convert KTable to a KStream using the previous value

2020-10-06 Thread Javier Freire Riobo
Hi all, I'd like to propose these changes to the Kafka Streams API. https://cwiki.apache.org/confluence/display/KAFKA/KIP-675%3A+Convert+KTable+to+a+KStream+using+the+previous+value This is a proposal to convert a KTable to a KStream knowing the previous value of the registry. I also opened a p

[jira] [Created] (KAFKA-10578) KIP-675: Convert KTable to a KStream using the previous value

2020-10-06 Thread Jira
Javier Freire Riobó created KAFKA-10578: --- Summary: KIP-675: Convert KTable to a KStream using the previous value Key: KAFKA-10578 URL: https://issues.apache.org/jira/browse/KAFKA-10578 Project