Hi all,
I recently had an experience where a streaming pipeline became "clogged"
due to invalid data reaching the final step in my pipeline such that the
data was causing a non-transient error when writing to my Sink. Since the
job is a streaming job, the element (bundle) was continuously retryin
Thanks for your responses Luke. One point I have confusion over:
* Modify the sink implementation to do what you want with the bad data and
> update the pipeline.
>
I modified the sink implementation to ignore the specific error that was
the problem and updated the pipeline. The update succeeded
>
> It is likely that the incorrect transform was edited...
>
It appears you're right; I tried to reproduce but this time was able to
clear the issue by making "the same" code change and updating the
pipeline. I believe it was just a change in the wrong place in code.
Good to know this works! T
