Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-02-11 Thread Sushil Ks
Thanks, Raghu. On Tue, Feb 6, 2018 at 6:41 AM, Raghu Angadi wrote: > Hi Sushil, > > That is expected behavior. If you don't have any saved checkpoint, the > pipeline would start from scratch. It does not have any connection to > previous run. > > On Thu, Feb 1, 2018 at

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-02-01 Thread Sushil Ks
; in which case KafkaIO would read from latest offset. There is probably some >>> way to verify if pipeline is restarting from a checkpoint. >>> >>> On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks wrote: >>> >>>> HI Aljoscha, >>>>

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-01-08 Thread Sushil Ks
PM, "Reuven Lax" wrote: > Do you set ENABLE_AUTO_COMMIT_CONFIG? > > On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks wrote: > >> HI Aljoscha, >>The issue is let's say I consumed 100 elements in 5 >> mins Fixed Window with *GroupByKey*

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-01-07 Thread Sushil Ks
s skipping the rest 30 elements. Wanted to know if this is expected? In case if you still having doubt let me know will share a code snippet. Regards, Sushil Ks

Re: KafkaIO reading from latest offset when pipeline fails on FlinkRunner

2018-01-04 Thread Sushil Ks
*bump* On Dec 15, 2017 11:22 PM, "Lukasz Cwik" wrote: > +dev@beam.apache.org > > On Thu, Dec 14, 2017 at 11:27 PM, Sushil Ks wrote: > >> Hi Likasz, >>I am not sure whether I can reproduce in the DirectRunner, as >> am taking retr