Re: Enable optional fields in csv parser
A better way of doing this would be to use the fieldTransformation setting and the REMOVE method to get rid of the extraneous fields. Docs are included at https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html# That way you don’t need a separate preprocessing step. Simon On Sat, 16 Nov 2019 at 16:09, Hema malini wrote: > Thanks ..will do preprocessing of data.. > > On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, > wrote: > >> No, there is no way to do this currently. >> >> The parser parses the line into and array of strings that must match the >> size of the columns. >> >> The underlying opencsv parser does not support this either. You may have >> to do some normalization work on your data if you need to account for this. >> >> >> >> >> On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) >> wrote: >> >> Hi all, >> >> Is there any way to mark some columns as optional in column mapping in >> CSV parser. >> >> Thanks and Regards, >> Hema >> >> -- -- simon elliston ball @sireb
Re: CSV parser
Hi ,
Thanks for your reply. Issue got resolved after several restarts. I faced
issues like when it is a new index it is taking time to index the data.
After restarting to nearly five times, suddenly data appeared in the index.
Thanks and Regards,
Hema
On Wed, 13 Nov, 2019, 12:48 PM Hema malini, wrote:
> Simon,
>
> Please find the sample config file
>
> {
> "parserClassName":"org.apache.metron.parsers.csv.CSVParser",
> "sensorTopic":"CSV",
> "parserConfig": {
> "deviceTimeZone": "UTC",
> "separator":",",
> "columns":{ 'A' : 1, 'B' : 2}
> }
> }
> Thanks and Regards,
> Hema
>
> On Wed, 13 Nov, 2019, 3:56 AM Simon Elliston Ball, <
> si...@simonellistonball.com> wrote:
>
>> Perhaps you could post your config? You should have a dictionary in it
>> called columns which maps column name to index.
>>
>> Simon
>>
>> On Tue, 12 Nov 2019 at 16:05, Hema malini
>> wrote:
>>
>>> Yes. I uploaded as mentioned in the document.
>>>
>>> On Tue, 12 Nov, 2019, 9:31 PM Simon Elliston Ball, <
>>> si...@simonellistonball.com> wrote:
>>>
Did you upload your configs to zookeeper?
On Tue, 12 Nov 2019 at 16:00, Hema malini
wrote:
> I referred the document - .
> https://metron.apache.org/current-book/metron-platform/metron-parsers/index.html..
> I modified the file $METRON_HOME/config/zookeeper/parsers/csv.json. i am
> getting still column metadata not defined.please let me know what I am
> missing in this.
>
> On Tue, 12 Nov, 2019, 9:23 PM Simon Elliston Ball, <
> si...@simonellistonball.com> wrote:
>
>> You modify the column data in the parser config. I suggest checking
>> the docs for the csv parser.
>>
>> On Tue, 12 Nov 2019 at 15:51, Hema malini
>> wrote:
>>
>>> Hi,
>>>
>>> I enabled CSV parser and created Kafka topic for CSV. Where i need
>>> to configure column metadata. Similarly for json parser where i should
>>> enable the json key ,value. Do i need to modify the parser class and
>>> redeploy the jar again.
>>>
>>> Thanks and regards,
>>> Hema
>>>
>> --
>> --
>> simon elliston ball
>> @sireb
>>
> --
--
simon elliston ball
@sireb
>>> --
>> --
>> simon elliston ball
>> @sireb
>>
>
Re: Enable optional fields in csv parser
Thanks ..will do preprocessing of data.. On Sat, 16 Nov, 2019, 9:25 PM Otto Fowler, wrote: > No, there is no way to do this currently. > > The parser parses the line into and array of strings that must match the > size of the columns. > > The underlying opencsv parser does not support this either. You may have > to do some normalization work on your data if you need to account for this. > > > > > On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) > wrote: > > Hi all, > > Is there any way to mark some columns as optional in column mapping in CSV > parser. > > Thanks and Regards, > Hema > >
Re: Enable optional fields in csv parser
No, there is no way to do this currently. The parser parses the line into and array of strings that must match the size of the columns. The underlying opencsv parser does not support this either. You may have to do some normalization work on your data if you need to account for this. On November 16, 2019 at 08:49:36, Hema malini (nhemamalin...@gmail.com) wrote: Hi all, Is there any way to mark some columns as optional in column mapping in CSV parser. Thanks and Regards, Hema
Enable optional fields in csv parser
Hi all, Is there any way to mark some columns as optional in column mapping in CSV parser. Thanks and Regards, Hema
