Hey 2 cents. Would you know which part of pipeline is 'breaking'? If you see your pipeline as. Source ----->processor ---->sink then you could put it through a unit test. Create some source data and pass it through. If your processor is fine you could then build a small pipeline which just read from source. If the issue is writing to be in the sink - most of my problems - you will do similar Sorry to say something you already know..but I am debugging all my pipelines like this..I found this as the quickest way Hth
On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <[email protected]> wrote: > Hi, > > I have a pipeline running on Dataflow that is reading messages (already > published) to a Pub/Sub topic and save to BigQuery. I’m running into an > issue that after processing tons of millions of messages the pipeline get > stuck and stop reading more messages from Pub/Sub. How can I start > investigating to find what’s going on? >
