Hi Mustafa, great. Let me know if it worked after the changes
Best, Stefan On Wed, Feb 27, 2019 at 6:12 AM Mustafa Akmal <mustafa.ak...@abcdata.org> wrote: > Ok Thanks Stefan! I'll give it a try > Regards, > Mustafa > > On Feb 27 2019, at 7:10 pm, Stefan Kupstaitis-Dunkler < > stefan....@gmail.com> wrote: > > Hi Mustafa, > > Looking at your parser config, you've set these values within the > "rawMessageStrategyConfig" > > *"readMetadata": "true",* > *"mergeMetadata": "true"* > > However, there they will be ignored. Have a look at the same properties > just below the "*errorWriterClassName*" property. Both of them are set > to false. > You can remove those properties from the "rawMessageStrategyConfig" > section and set them true in the upper part of your config! I'm confident > that will make it work for you! > > Best, > Stefan > > [image: Sent from Mailspring] > On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <mustafa.ak...@abcdata.org> > wrote: > > Hi Stefan > Thanks for the quick response. > I am using Metron version 0.6.0.1.7.1.0 > <https://link.getmailspring.com/link/750f1435-c1ca-4f45-b67f-3a275105c...@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > Here is my parser config > { > "parserClassName": "org.apache.metron.parsers.csv.CSVParser", > "filterClassName": null, > "sensorTopic": "test_syslog", > "outputTopic": null, > "errorTopic": null, > "writerClassName": null, > "errorWriterClassName": null, > "readMetadata": false, > "mergeMetadata": false, > "numWorkers": null, > "numAckers": null, > "spoutParallelism": 1, > "spoutNumTasks": 1, > "parserParallelism": 1, > "parserNumTasks": 1, > "errorWriterParallelism": 1, > "errorWriterNumTasks": 1, > "spoutConfig": {}, > "securityProtocol": null, > "stormConfig": {}, > "parserConfig": { > "columns": { > "Id": 0, > "Name": 1, > "Age": 2 > } > }, > "fieldTransformations": [], > "cacheConfig": {}, > "rawMessageStrategy": "ENVELOPE", > "rawMessageStrategyConfig": { > "messageField": "data", > "readMetadata": "true", > "mergeMetadata": "true" > } > } > > I have been trying to do this for a couple of days. Tried it on multiple > different parsers after studying documentation. At first I was using a > parser chaining usecase. But now I wrote a simple parser to test the > problem. Set the readMetadata and mergeMetadata fields to true just in > case. It just doesn't work. > Regards, > Mustafa Akmal > > On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler < > stefan....@gmail.com> wrote: > > Hi Mustafa, > > > - can you verify if the "*mergeMetaData*" property in the parser json > is set to "*true*". > - If this property is set to false, other fields won't be merged as > meta data. > - If this property is not set, it should default to true for the > message strategy "ENVELOPE". > - Any other behaviour is probably a bug. > - Also verify if you set "*rawMessageStrategy*" to "*true*" for the > same reasons. > > Best, > Stefan > > [image: Sent from Mailspring] > On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mustafa.ak...@abcdata.org> > wrote: > > Hello > I am using a CSV parser. I have the following log > {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"} > > Now I have set the raw message strategy to 'ENVELOPE' and the messageField > to 'data' > However after the record is indexed in elasticsearch, the parser does > parse the value inside data but it does not get the additional fields as > shown in the original log that is 'AdditionField' and 'AdditionField2'. > What am I doing wrong? Can anyone help? > Thanks! > > [image: Sent from Mailspring] > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > > > > -- > Stefan Kupstaitis-Dunkler > https://datahovel.com/ > <https://link.getmailspring.com/link/750f1435-c1ca-4f45-b67f-3a275105c...@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > https://twitter.com/StefanDunkler > <https://link.getmailspring.com/link/750f1435-c1ca-4f45-b67f-3a275105c...@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > > > > -- > Stefan Kupstaitis-Dunkler > https://datahovel.com/ > <https://link.getmailspring.com/link/cb6afda8-662c-4388-8b27-f4cee1406...@getmailspring.com/0?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > https://twitter.com/StefanDunkler > <https://link.getmailspring.com/link/cb6afda8-662c-4388-8b27-f4cee1406...@getmailspring.com/1?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > <#m_-8837950766558748118_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > -- Stefan Kupstaitis-Dunkler https://datahovel.com/ https://twitter.com/StefanDunkler
