I also have a similar use case (but with BigTable) that I feel like I had to hack up to make work. It'd be great to hear if there is a way to do something like this already, or if there are plans in the future.
On Fri, Aug 25, 2017 at 9:46 AM, Chaim Turkel <[email protected]> wrote: > Hi, > I have a few piplines that are an ETL from different systems to bigquery. > I would like to write the status of the ETL after all records have > been updated to the bigquery. > The problem is that writing to bigquery is a sink and you cannot have > any other steps after the sink. > I tried a sideoutput, but this is called in no correlation to the > writing to bigquery, so i don't know if it succeeded or failed. > > > any ideas? > chaim >
