How often does your pipeline checkpoint/snapshot? If the failure happens before the first checkpoint, the pipeline could restart without any state, in which case KafkaIO would read from latest offset. There is probably some way to verify if pipeline is restarting from a checkpoint.
On Sun, Jan 7, 2018 at 10:57 PM, Sushil Ks <sushil...@gmail.com> wrote: > HI Aljoscha, > The issue is let's say I consumed 100 elements in 5 > mins Fixed Window with *GroupByKey* and later I applied *ParDO* for all > those elements. If there is an issue while processing element 70 in > *ParDo *and the pipeline restarts with *UserCodeException *it's skipping > the rest 30 elements. Wanted to know if this is expected? In case if you > still having doubt let me know will share a code snippet. > > Regards, > Sushil Ks >
