Re: Enable optional fields in csv parser
I think what he is saying is that the csv files may not always have all the columns, in which case they won’t parse, which is before stellar can do anything. On November 16, 2019 at 11:30:25, Simon Elliston Ball ( si...@simonellistonball.com) wrote: A better way of doing this would be to use the fieldTransformation setting and the REMOVE method to get rid of the extraneous fields. Docs are included at https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html# That way you don’t need a separate preprocessing step. Simon On Sat, 16 Nov 2019 at 16:09, Hema malini wrote: > Thanks ..will do preprocessing of data.. > > On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, > wrote: > >> No, there is no way to do this currently. >> >> The parser parses the line into and array of strings that must match the >> size of the columns. >> >> The underlying opencsv parser does not support this either. You may have >> to do some normalization work on your data if you need to account for this. >> >> >> >> On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) >> wrote: >> >> Hi all, >> >> Is there any way to mark some columns as optional in column mapping in >> CSV parser. >> >> Thanks and Regards, >> Hema >> >> -- -- simon elliston ball @sireb
Re: Enable optional fields in csv parser
A better way of doing this would be to use the fieldTransformation setting and the REMOVE method to get rid of the extraneous fields. Docs are included at https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html# That way you don’t need a separate preprocessing step. Simon On Sat, 16 Nov 2019 at 16:09, Hema malini wrote: > Thanks ..will do preprocessing of data.. > > On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, > wrote: > >> No, there is no way to do this currently. >> >> The parser parses the line into and array of strings that must match the >> size of the columns. >> >> The underlying opencsv parser does not support this either. You may have >> to do some normalization work on your data if you need to account for this. >> >> >> >> >> On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) >> wrote: >> >> Hi all, >> >> Is there any way to mark some columns as optional in column mapping in >> CSV parser. >> >> Thanks and Regards, >> Hema >> >> -- -- simon elliston ball @sireb
Re: Enable optional fields in csv parser
Thanks ..will do preprocessing of data.. On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, wrote: > No, there is no way to do this currently. > > The parser parses the line into and array of strings that must match the > size of the columns. > > The underlying opencsv parser does not support this either. You may have > to do some normalization work on your data if you need to account for this. > > > > > On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) > wrote: > > Hi all, > > Is there any way to mark some columns as optional in column mapping in CSV > parser. > > Thanks and Regards, > Hema > >
Re: Enable optional fields in csv parser
No, there is no way to do this currently. The parser parses the line into and array of strings that must match the size of the columns. The underlying opencsv parser does not support this either. You may have to do some normalization work on your data if you need to account for this. On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) wrote: Hi all, Is there any way to mark some columns as optional in column mapping in CSV parser. Thanks and Regards, Hema
Enable optional fields in csv parser
Hi all, Is there any way to mark some columns as optional in column mapping in CSV parser. Thanks and Regards, Hema
