Hi Mustafa, Looking at your parser config, you've set these values within the "rawMessageStrategyConfig"
*"readMetadata": "true",* *"mergeMetadata": "true"* However, there they will be ignored. Have a look at the same properties just below the "*errorWriterClassName*" property. Both of them are set to false. You can remove those properties from the "rawMessageStrategyConfig" section and set them true in the upper part of your config! I'm confident that will make it work for you! Best, Stefan On Wed, Feb 27, 2019 at 6:04 AM Mustafa Akmal <mustafa.ak...@abcdata.org> wrote: > Hi Stefan > Thanks for the quick response. > I am using Metron version 0.6.0.1.7.1.0 > <https://link.getmailspring.com/link/750f1435-c1ca-4f45-b67f-3a275105c...@getmailspring.com/0?redirect=0.6.0.1.7.1.0&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > Here is my parser config > { > "parserClassName": "org.apache.metron.parsers.csv.CSVParser", > "filterClassName": null, > "sensorTopic": "test_syslog", > "outputTopic": null, > "errorTopic": null, > "writerClassName": null, > "errorWriterClassName": null, > "readMetadata": false, > "mergeMetadata": false, > "numWorkers": null, > "numAckers": null, > "spoutParallelism": 1, > "spoutNumTasks": 1, > "parserParallelism": 1, > "parserNumTasks": 1, > "errorWriterParallelism": 1, > "errorWriterNumTasks": 1, > "spoutConfig": {}, > "securityProtocol": null, > "stormConfig": {}, > "parserConfig": { > "columns": { > "Id": 0, > "Name": 1, > "Age": 2 > } > }, > "fieldTransformations": [], > "cacheConfig": {}, > "rawMessageStrategy": "ENVELOPE", > "rawMessageStrategyConfig": { > "messageField": "data", > "readMetadata": "true", > "mergeMetadata": "true" > } > } > > I have been trying to do this for a couple of days. Tried it on multiple > different parsers after studying documentation. At first I was using a > parser chaining usecase. But now I wrote a simple parser to test the > problem. Set the readMetadata and mergeMetadata fields to true just in > case. It just doesn't work. > Regards, > Mustafa Akmal > > On Feb 27 2019, at 6:49 pm, Stefan Kupstaitis-Dunkler < > stefan....@gmail.com> wrote: > > Hi Mustafa, > > > - can you verify if the "*mergeMetaData*" property in the parser json > is set to "*true*". > - If this property is set to false, other fields won't be merged as > meta data. > - If this property is not set, it should default to true for the > message strategy "ENVELOPE". > - Any other behaviour is probably a bug. > - Also verify if you set "*rawMessageStrategy*" to "*true*" for the > same reasons. > > Best, > Stefan > > [image: Sent from Mailspring] > On Wed, Feb 27, 2019 at 5:34 AM Mustafa Akmal <mustafa.ak...@abcdata.org> > wrote: > > Hello > I am using a CSV parser. I have the following log > {"data": "1,john,23","AdditionField": "abcd","AdditionField2": "12345"} > > Now I have set the raw message strategy to 'ENVELOPE' and the messageField > to 'data' > However after the record is indexed in elasticsearch, the parser does > parse the value inside data but it does not get the additional fields as > shown in the original log that is 'AdditionField' and 'AdditionField2'. > What am I doing wrong? Can anyone help? > Thanks! > > [image: Sent from Mailspring] > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > > > > -- > Stefan Kupstaitis-Dunkler > https://datahovel.com/ > <https://link.getmailspring.com/link/750f1435-c1ca-4f45-b67f-3a275105c...@getmailspring.com/1?redirect=https%3A%2F%2Fdatahovel.com%2F&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > https://twitter.com/StefanDunkler > <https://link.getmailspring.com/link/750f1435-c1ca-4f45-b67f-3a275105c...@getmailspring.com/2?redirect=https%3A%2F%2Ftwitter.com%2FStefanDunkler&recipient=dXNlckBtZXRyb24uYXBhY2hlLm9yZw%3D%3D> > > > > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > Virus-free. > www.avg.com > <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > <#m_-7107511858632782524_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> > -- Stefan Kupstaitis-Dunkler https://datahovel.com/ https://twitter.com/StefanDunkler
