Thanks Matt. Validaterecord would be useful. I am using SplitRecord to create batches for the large input files for efficiency of processing.
On Mon, Aug 13, 2018 at 6:02 PM, Matt Burgess <mattyb...@apache.org> wrote: > You can use ValidateRecord (with a CSVReader and JSONRecordSetWriter, > and another "invalid CSV Reader" for invalid records) for that, then > SplitRecord if you need it. However if you can describe your > downstream flow, perhaps we can help you avoid the need to split the > records at all (unless you are using a downstream processor that only > handles one record/JSON object at a time). > > Regards, > Matt > > On Mon, Aug 13, 2018 at 7:41 AM saloni udani <saloniudani...@gmail.com> > wrote: > > > > Hi > > > > I have a bunch of CSV files which I need to convert to JSON. > > My current flow is > > > > GetFile --> SplitRecord (CSVReader and JSONRecordSetWriter) > > > > > > The issue is if the csv contains an invalid records then the file gets > stuck in the queue. Is there a way to discard the invalid CSV lines > encountered to failure relationship? Documentation says that only those > records will be routed to failure which fails csv-->json conversion. But > here I want even the invalid csv records to be routed to failure. > > > > > > Thanks > > Saloni Udani >
