Thanks, I didn’t try this yet.
It’s just weird to see the throughput decreasing from millions of rows per
second to just a few hundred per second.
Em 27 de out. de 2020 10:03 -0300, Sofia’s World <[email protected]> escreveu:
> Hey
> 2 cents. Would you know which part of pipeline is 'breaking'? If you see
> your pipeline as.
> Source ----->processor ---->sink then you could put it through a unit test.
> Create some source data and pass it through.
> If your processor is fine you could then build a small pipeline which just
> read from source.
> If the issue is writing to be in the sink - most of my problems - you will do
> similar
> Sorry to say something you already know..but I am debugging all my pipelines
> like this..I found this as the quickest way
> Hth
>
> > On Tue, Oct 27, 2020, 12:30 PM Thiago Chiarato <[email protected]> wrote:
> > > Hi,
> > >
> > > I have a pipeline running on Dataflow that is reading messages (already
> > > published) to a Pub/Sub topic and save to BigQuery. I’m running into an
> > > issue that after processing tons of millions of messages the pipeline get
> > > stuck and stop reading more messages from Pub/Sub. How can I start
> > > investigating to find what’s going on?
