Hey guys,
I have a question, does withkeys transformation enforce a reshuffle?
My pipeline basically look like this PubsubLiteIO -> ParDo(..) -> ParDo()
-> BigqueryIO.write()
The problem is PubsubLiteIO -> ParDo(..) -> ParDo() always fused together.
But The ParDo is expensive and I want
Some comments here:
1. All messages in a PubSub topic is not a well-defined statement, as
there can always be more messages published. You may know that nobody will
publish any more messages, but the pipeline does not.
2. While it's possible to read from Pub/Sub in batch, it's usually not
Hi all,
I want to create a Dataflow pipeline using Pub/sub as an input connector
but I want to run it in batch mode and not streaming mode. I know it's not
possible in Python but how can I achieve this in Java? Basically, I want my
pipeline to read all messages in a Pubsub topic, process and
