As seen in this image https://imgur.com/a/wrZET97, I'm trying to deal with late
data (intentionally stopped my consumer so data has been accumulating for
several days now). Now, with the following Window... I'm using Beam 2.27 and
Flink 1.12.
Window.into(FixedWindows.of(Duration.standardMinutes(10)))
And several parsing stages after, once it's time to write within the ParquetIO
stage...
FileIO
.<String, MyClass>writeDynamic()
.by(...)
.via(...)
.to(...)
.withNaming(...)
.withDestinationCoder(StringUtf8Coder.of())
.withNumShards(options.getNumShards())
it won't send bytes across all stages so no data is being written, still it
accumulates in the first stage seen in the image and won't go further than that.
Any reason why this may be happening? Wrong windowing strategy?