Right. Try to swap BQ with instead writing to bucket. I am wondering if some of your messages result in invalid bq structure.. Are you swallowing exception by chance? Have seen similar behavior when writing to Bq. With millions of msgs it's hard to debug. Perhaps someone here can suggest how to debug busted writes to Bq? Kr
On Tue, Oct 27, 2020, 1:32 PM Thiago Chiarato <[email protected]> wrote: > Thanks, I didn’t try this yet. > It’s just weird to see the throughput decreasing from millions of rows per > second to just a few hundred per second. > Em 27 de out. de 2020 10:03 -0300, Sofia’s World <[email protected]> > escreveu: > > Hey > 2 cents. Would you know which part of pipeline is 'breaking'? If you see > your pipeline as. > Source ----->processor ---->sink then you could put it through a unit > test. Create some source data and pass it through. > If your processor is fine you could then build a small pipeline which just > read from source. > If the issue is writing to be in the sink - most of my problems - you will > do similar > Sorry to say something you already know..but I am debugging all my > pipelines like this..I found this as the quickest way > Hth > > On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <[email protected]> wrote: > >> Hi, >> >> I have a pipeline running on Dataflow that is reading messages (already >> published) to a Pub/Sub topic and save to BigQuery. I’m running into an >> issue that after processing tons of millions of messages the pipeline get >> stuck and stop reading more messages from Pub/Sub. How can I start >> investigating to find what’s going on? >> >
