Re: KStreams Proposal

2016-03-14 Thread Guozhang Wang
Thanks Bill for the detailed description of your use case. I think part of the issue is that today KStream does not allow non-keyed aggregation, and hence users need to call a `map` function if they want to do aggregations on other fields. We did this by design for KStream since if users write the

Re: KStreams Proposal

2016-03-13 Thread Bill Bejeck
Hi Guozhang, Possibly, but the use case I'm working with is having a of collector object , for aggregate statistics for example, that would output results intermittently (via punctuate). The issue for me is that 'transform(..)' returns a key-value pair for each message, possibly of a different

Re: KStreams Proposal

2016-03-13 Thread Guozhang Wang
Hello Bill, We added transform() together with process() to support any user-customized stateful processor that can still concatenate to another KStream. So for your case, would `transform(...).to(topic)` provide the same functionality as "processTo(topic, ...)"? Guozhang On Sat, Mar 12, 2016

KStreams Proposal

2016-03-12 Thread Bill Bejeck
Hi All, While working with KStream/KStreamImp I discovered that there does not seem to be any way to connect the results of the KStream.process method with a sink node. I'd like to propose an addition to the API a "processTo" method. I've looked at and used the "transform", "reduceByKey" and "ag