It sounds interesting. Thanks ! Regards JB
On 03/10/2018 01:49, Jeff Klukas wrote: > I've seen a few Beam users mention the need to handle errors in their > transforms by using a try/catch and routing to different outputs based > on whether an exception was thrown. This was particularly nicely written > up in a post by Vallery Lancey: > > https://medium.com/@vallerylancey/error-handling-elements-in-apache-beam-pipelines-fffdea91af2a > > I'd love to see this pattern better supported directly in the Beam API, > because it currently requires the user to implement a full DoFn even for > the simplest cases. > > I propose we support for a MapElements-like transform that allows the > user to specify a set of exceptions to catch and route to a failure > output. Something like: > > MapElements > .via(myFunctionThatThrows) > .withSuccessTag(successTag) > .withFailureTag(failureTag, JsonParsingException.class) > > which would output a PCollectionTuple with both the successful outcomes > of the map operation and also a collection of the inputs that threw > JsonParsingException. > > To make this more concrete, I put together a proof of concept PR: > https://github.com/apache/beam/pull/6518 I'd appreciate feedback about > whether this seems like a worthwhile addition and a feasible approach. -- Jean-Baptiste Onofré [email protected] http://blog.nanthrax.net Talend - http://www.talend.com
