I think the simplest solution would be to have some kind of override/hook that allows Flink/Spark/... to provide storage. They already have a concept of a job and know how to store them so can we piggyback the Beam pipeline there.
On Fri, Jun 28, 2019 at 7:49 AM Chad Dombrova <[email protected]> wrote: > > In reality, a more complex job service is needed that is backed by some >> kind of persistent storage or stateful service. >> > > I was afraid you were going to say that :) So is anything like this > planned or in the works? > > I see that there's also ReferenceRunnerJobService, but it seems like a > very similar implementation to InMemoryJobService. What's the use case for > that? > > -chad > >
