I just watched the excellent presentation by Markku Leppisto in Singapore.
I consult for a financial broker in London.
Our use case is streaming financial data on which various analytics are
performed to find relative value trading opportunities in the fixed income
markets.
I have two questions about Apache Beam:
a) Does it, or will it, support dynamic topologies? We have many analysts
all of whom want slight variations of the topologies, and they may want to
change the topologies, or the function ("node") parameters, at runtime. Is
this possible, and if not is "second prize" of simply growing the DAG with
non-recombinant node branches at runtime, possible?
b) Does, or will Apache beam support "replays"? We find quite often that
historical inputs change, a long time later (for example, a bond price from
a few weeks ago is seen to have been erroneous, and is changed). We then
have to re-run from that point forward as all downstream calcs are
dependent on all prices. I believe Apache Flink supports this type of
functionality? Is this possilble with Beam?
I am evaluating the various dataflow programming environments that are
springing up and will advise based partly on the above.
Many thanks,
Thomas
