Re: KStreams Rewind Offset

2016-06-03 Thread Guozhang Wang
Hello Michael, Just want to clarify a few thing before I can provide some suggestions in simplifying your pipeline with Kafka Streams: as for the "race condition between reading the stream data and the data being removed", do you mean that the stream records is actually mutable, hence after they a

Re: KStreams Rewind Offset

2016-06-02 Thread Michael D. Coon
Mattias,    That's disappointing given that Kafka offers me the ability to rewind and replay data. My use case is that we are building graph data structures based on data indexed from a live stream. At any time, the live data content may be marked for deletion for any number of reasons; but duri

Re: KStreams Rewind Offset

2016-06-02 Thread Matthias J. Sax
Hi Mike, currently, this is not possible. We are already discussing some changes with regard to reprocess. However, I doubt that going back to a specific offset of a specific partition will be supported as it would be too difficult to reset the internal data structures and intermediate results cor

KStreams Rewind Offset

2016-06-01 Thread Michael D. Coon
All,   I think it's great that the ProcessorContext offers the partition and offset of the current record being processed; however, it offers no way for me to actually use the information. I would like to be able to rewind to a particular offset on a partition if I needed to. The consumer is als