Hi all,
I am using a PubSubIO source, windowing every 10 seconds and then doing
something with the windows, for example:
pipeline
.apply(PubsubIO.readPubsubMessagesWithAttributes().fromSubscription(..))
.apply(Window.into(FixedWindows.of(Duration.standardSeconds(10))))
.apply(MapElements
.into(TypeDescriptors.strings())
.via((PubsubMessage msg) -> msg.getAttribute(*..*)))
.apply(new WriteOneFilePerWindow(..));
My expectation was that if I publish a pubsub message, and then
publish another 10+ seconds later, a single file should be written for
the previous 10 second window. However I find that I need to publish a
lot of messages for any files to be written at all (e.g. 30+
messages).
Is this expected behaviour when using PubSubIO? Is there a way to
tweak it to fire the windows more eagerly?
Thanks,
Josh