Hello Beam list!
We are looking at adopting some more advanced use cases with Beam code at
its core including automated testing and data dependency tracking.
Specifically I'm interested in things like making sure data changes don't
break pipelines, or things that depend on pipeline output, especi
Hi Charles,
Maybe you can setup data sets and use the TestPipeline to validate (with
PAssert) that it works as expected in your pipeline.
The data sets can be store somewhere (database or filesystem) and loaded in
tests (basically as we do in the Beam ITs).
Thought ?
Regards
JB
On 01/03/2
Take a look at the drain, update and inplace snapshot proposal threads,
they make it easier to address some of your problems.
https://lists.apache.org/thread.html/37c9aa4aa5011801a7f060bf3f53e687e539fa6154cc9f1c544d4f7a@%3Cdev.beam.apache.org%3E
https://lists.apache.org/thread.html/3cfbd650a46327af
