Deploy exact copy of the job aside, separate subscriptions, buckets and destination datasets. If it will work smoothly it could be an internal dataflow issue.
Have you contacted Google support? They have more tools and metrics for dataflow job investigation than available on GCP console :) Regards, Marcin > Wiadomość napisana przez Sofia’s World <[email protected]> w dniu > 27.10.2020, o godz. 14:54: > > > Right. Try to swap BQ with instead writing to bucket. I am wondering if some > of your messages result in invalid bq structure.. > Are you swallowing exception by chance? Have seen similar behavior when > writing to Bq. > With millions of msgs it's hard to debug. Perhaps someone here can suggest > how to debug busted writes to Bq? > Kr > >> On Tue, Oct 27, 2020, 1:32 PM Thiago Chiarato <[email protected]> wrote: >> Thanks, I didn’t try this yet. >> It’s just weird to see the throughput decreasing from millions of rows per >> second to just a few hundred per second. >> Em 27 de out. de 2020 10:03 -0300, Sofia’s World <[email protected]> >> escreveu: >>> Hey >>> 2 cents. Would you know which part of pipeline is 'breaking'? If you see >>> your pipeline as. >>> Source ----->processor ---->sink then you could put it through a unit test. >>> Create some source data and pass it through. >>> If your processor is fine you could then build a small pipeline which just >>> read from source. >>> If the issue is writing to be in the sink - most of my problems - you will >>> do similar >>> Sorry to say something you already know..but I am debugging all my >>> pipelines like this..I found this as the quickest way >>> Hth >>> >>>> On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <[email protected]> wrote: >>>> Hi, >>>> >>>> I have a pipeline running on Dataflow that is reading messages (already >>>> published) to a Pub/Sub topic and save to BigQuery. I’m running into an >>>> issue that after processing tons of millions of messages the pipeline get >>>> stuck and stop reading more messages from Pub/Sub. How can I start >>>> investigating to find what’s going on?
