Hey Josh, There isn't really generic functionality for this as we don't want to make "data loss" easy. There are some ongoing designs for specific transforms (e.g., BEAM-190 for BigQueryIO). One easy thing to do in this case might be to wrap the code in a try/catch and if you catch an exception then return some table name like "leftovers".
Dan On Tue, May 16, 2017 at 8:02 AM, Josh <[email protected]> wrote: > Hi all, > > I am wondering if there is there a way to make Beam skip certain failures > - for example I am using BigQueryIO to write to a table, where the table > name is chosen dynamically: > > > ``` > > .apply(BigQueryIO.<TableRow>write() > > .to(new ExtractTableName())) > > ``` > > > I want to make it so that, if for some reason my ExtractTableName instance > (which is a SerializableFunction<ValueInSingleWindow<TableRow>, > TableDestination>) throws an exception, then the exception is logged and > the write is skipped. > > > Is it possible to achieve this behaviour without modifying the Beam > codebase/BigQueryIO retry logic? > > At the moment if my function throws an exception, the write is retried > indefinitely. > > > Thanks, > > Josh >
