OK, nevermind, I figured it out. The issue was that the FIRST argument to withOutputTags() has to have the same generic type as the DoFn's output. So by changing my transform to a DoFn<FileIO.ReadableFile, ErrorInfo> I can successfully mix tags.
From: Kelsey RIDER <kelsey.ri...@ineat-conseil.fr> Sent: jeudi 26 juillet 2018 09:45 To: user@beam.apache.org Subject: Pipeline error handling I'm trying to figure out how to handle errors in my Pipeline. Right now, my main transform is a DoFn<FileIO.ReadableFile, CSVRecord>. I have a few different TupleTag<CSVRecord> that I use depending on the data contained in the records. In the event there's a problem with a line (due to one of several possible causes), I created a TupleTag<CSVRecord> ERROR. However, just doing this doesn't carry with it any information about the error. I would like for the ERROR tag to have a type other than CSVRecord, e.g. some sort of ErrorInfo class containing the row number, filename, message about what went wrong, etc... I can't use multiple TupleTag types with ParDo, because the withOutputTags() method forces them to all have the same generic parameter. I saw the example here: https://medium.com/@vallerylancey/error-handling-elements-in-apache-beam-pipelines-fffdea91af2a But I don't see how this can work, since they use multiple generic types in withOutputTags(). (And is this good practice? Seems like they "cheat" by not calling apply(), instead directly transforming the PCollection (and why even bother extending DoFn in this case?).) Finally, if I write my own PTransform<FileIO.ReadableFile, PCollectionTuple> class, and start manually creating PCollections and whatnot...then this would effectively become a bottleneck where everything has to be read at once, and there's no longer any sequential handling of the records as they're read, right? Suite à l'évolution des dispositifs de réglementation du travail, si vous recevez ce mail avant 7h00, en soirée, durant le week-end ou vos congés merci, sauf cas d'urgence exceptionnelle, de ne pas le traiter ni d'y répondre immédiatement. Suite à l'évolution des dispositifs de réglementation du travail, si vous recevez ce mail avant 7h00, en soirée, durant le week-end ou vos congés merci, sauf cas d'urgence exceptionnelle, de ne pas le traiter ni d'y répondre immédiatement.
