Hello Everyone
I have a question, I have on my pipeline one side input that fetches some
configurations from an API endpoint each 30 seconds, my question is this.
I have something similar to what is showed in the side input patterns
documentation
PCollectionView<Map<String, String>> map =
p.apply(GenerateSequence.from(0).withRate(1,
Duration.standardSeconds(5L)))
.apply(
ParDo.of(
new DoFn<Long, Map<String, String>>() {
@ProcessElement
public void process(
@Element Long input,
@Timestamp Instant timestamp,
OutputReceiver<Map<String, String>> o) {
call HTTP endpoint here!!
}
}))
.apply(
Window.<Map<String, String>>into(new GlobalWindows())
.triggering(Repeatedly.forever(AfterProcessingTime.pastFirstElementInPane()))
.discardingFiredPanes())
.apply(Latest.globally())
.apply(View.asSingleton());
What happens if for example the HTTP endpoint takes time to respond due
some network issues and/or the amount of data. Is this gonna introduce
delays on my main pipeline? Is the main pipeline blocked until the pardo
that processes the side input ends?
I don't care too much about the consistency here, I mean if the
configuration changed in the Time T1 I don't care if some registries with
T2 timestamp are processed with the configuration version of T1.
Regards.