[
https://issues.apache.org/jira/browse/METRON-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408344#comment-16408344
]
ASF GitHub Bot commented on METRON-1496:
----------------------------------------
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/969
> Stateful parsing: In one of our to-be-written parsers, we need to read an
external file, parse the file and keep the parsed file in memory. As far as I
can see, Stellar is stateless and not capable of keeping data in memory. It
might be possible by using environment variables, but this approach might get
messy.
With the heavy caching we use I wouldn't expect this to be a problem.
There is also a function like `OBJECT_GET` that allows you to specifically
customize a backing cache. This seems to fit the use case that you are
describing, but I could be wrong.
https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#object_get
> ChainLink Parser to reuse parser code at parserConfig level
> -----------------------------------------------------------
>
> Key: METRON-1496
> URL: https://issues.apache.org/jira/browse/METRON-1496
> Project: Metron
> Issue Type: Improvement
> Reporter: Bas van de Lustgraaf
> Priority: Minor
>
> During the development of some custom parsers we wrote a couple of classes /
> functions to make it possible to reuse code and assemble parser quicker at
> java coding level.
> We took this idea one step further and created the so called ChainLinkParser.
> This parser gives user without any java knowledge the opportunity to assemble
> parsers at parser configuration level.
> We would like to discuss the code and see if it can be submitted to the
> project. We will create a PR during this week to submit the code for review
> and discussion.
> Below you'll find an example of our Parser configuration for Suricata, which
> is using our ChainParser.
>
> {noformat}
> {
> "parserClassName":"nl.qsight.chainparser.ChainParser",
> "sensorTopic":"suricata",
> "readMetadata":true,
> "mergeMetadata":true,
> "numWorkers":3,
> "numAckers":3,
> "spoutParallelism":6,
> "spoutNumTasks":6,
> "parserParallelism":20,
> "parserNumTasks":20,
> "errorWriterParallelism":1,
> "errorWriterNumTasks":1,
> "spoutConfig":{
> "spout.firstPollOffsetStrategy":"LATEST"
> },
> "stormConfig":{
> "topology.max.spout.pending":2000
> },
> "parserConfig":{
> "chain":[
> "parse_json",
> "parse_username",
> "rename_fields",
> "parse_datetime"
> ],
> "parsers":{
> "parse_json":{
> "class":"nl.qsight.links.io.JSONDecoderLink"
> },
> "parse_username":{
> "class":"nl.qsight.links.io.RegexLink",
> "pattern":"(?i)(user|username|log)[=:](\\w+)",
> "selector":{
> "username":"2"
> },
> "input":"{{payload_printable}}"
> },
> "rename_fields":{
> "class":"nl.qsight.links.fields.RenameLink",
> "rename":{
> "proto":"protocol",
> "dest_ip":"ip_dst_addr",
> "src_ip":"ip_src_addr",
> "dest_port":"ip_dst_port",
> "src_port":"ip_src_port"
> }
> },
> "parse_datetime":{
> "class":"nl.qsight.links.io.TimestampLink",
> "patterns":[
> [
>
> "([0-9]{4})-([0-9]+)-([0-9]+)T([0-9]+):([0-9]+):([0-9]+).([0-9]+)([+-]{1}[0-9]{1,2}[:]?[0-9]{2})",
> "yyyy MM dd HH mm ss SSSSSS Z",
> "newest"
> ]
> ],
> "input":"{{timestamp}}"
> }
> }
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
