Re: Data (re)processing with Kafka (new wiki page)
Thanks a lot, Matthias! I have already begun to provide feedback. -Michael On Wed, Nov 23, 2016 at 11:41 PM, Matthias J. Sax wrote: > Hi, > > we added a new wiki page that is supposed to collect data (re)processing > scenario with Kafka: > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+Data+(Re) > Processing+Scenarios > > We added already a couple of scenarios we think might be common and want > to invite all of you to add more. This helps to get a better overview of > requirements to enable new use cases. > > We are looking forward to your feedback! > > > -Matthias > >
Re: Data (re)processing with Kafka (new wiki page)
This page is really helpful.Thanks for putting this Some nice to have features can be (not sure for this wiki page) 1) Pause and resume without having to start and stop. It should drain all the inflight calculations before doing the actual pause and a notifier will be helpful that it is actually paused. This can be much light wait if possible rather than stopping all tasks and stores and starting them again which might take a lot of time 2) Metadata api response if it can have all the topology graphs and sub graphs with all the nodes and edges for each sub graphs with corresponding state store names then it will be easier to build some UI on top of it and also the pipe between the sub graphs which are kafka topics need to be called out and also the time semantics can be laid out on top of it. This is something like a logical view on top of the physical view which the current api has. Regards Sai On Fri, Nov 25, 2016 at 12:53 AM, Michael Noll wrote: > Thanks a lot, Matthias! > > I have already begun to provide feedback. > > -Michael > > > > On Wed, Nov 23, 2016 at 11:41 PM, Matthias J. Sax > wrote: > > > Hi, > > > > we added a new wiki page that is supposed to collect data (re)processing > > scenario with Kafka: > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > Kafka+Streams+Data+(Re) > > Processing+Scenarios > > > > We added already a couple of scenarios we think might be common and want > > to invite all of you to add more. This helps to get a better overview of > > requirements to enable new use cases. > > > > We are looking forward to your feedback! > > > > > > -Matthias > > > > >
Re: Data (re)processing with Kafka (new wiki page)
Thanks for your input! For (2), there is currently KafkaStream.toString(). You could theoretically parse the String to extract all required information -- of course, this is not a nice solution. -Matthias On 11/25/16 2:11 AM, saiprasad mishra wrote: > This page is really helpful.Thanks for putting this > > Some nice to have features can be (not sure for this wiki page) > > 1) Pause and resume without having to start and stop. > It should drain all the inflight calculations before doing the actual pause > and a notifier will be helpful that it is actually paused. > This can be much light wait if possible rather than stopping all tasks and > stores and starting them again which might take a lot of time > > > 2) Metadata api response if it can have all the topology graphs and sub > graphs with all the nodes and edges for each sub graphs with corresponding > state store names then it will be easier to build some UI on top of it and > also the pipe between the sub graphs which are kafka topics need to be > called out and also the time semantics can be laid out on top of it. This > is something like a logical view on top of the physical view which the > current api has. > > > Regards > Sai > > On Fri, Nov 25, 2016 at 12:53 AM, Michael Noll wrote: > >> Thanks a lot, Matthias! >> >> I have already begun to provide feedback. >> >> -Michael >> >> >> >> On Wed, Nov 23, 2016 at 11:41 PM, Matthias J. Sax >> wrote: >> >>> Hi, >>> >>> we added a new wiki page that is supposed to collect data (re)processing >>> scenario with Kafka: >>> >>> https://cwiki.apache.org/confluence/display/KAFKA/ >> Kafka+Streams+Data+(Re) >>> Processing+Scenarios >>> >>> We added already a couple of scenarios we think might be common and want >>> to invite all of you to add more. This helps to get a better overview of >>> requirements to enable new use cases. >>> >>> We are looking forward to your feedback! >>> >>> >>> -Matthias >>> >>> >> > signature.asc Description: OpenPGP digital signature