[
https://issues.apache.org/jira/browse/METRON-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637336#comment-16637336
]
ASF GitHub Bot commented on METRON-1761:
----------------------------------------
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1184
@ottobackwards I see what you're saying. It looks like that could
definitely work. Thinking out loud here, but might that conflate the semantics
of our validation a bit? Validate currently does things like ensure that a
timestamp exists on the message, though I don't see why we couldn't expand it
to validations outside of our global Metron context.
One class that might be worth checking out is the unified enrichment
topology. This was changed to include a parallel enricher that handles errors
and message results in an EnrichmentResult class.
1.
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/UnifiedEnrichmentBolt.java#L270
2.
https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java#L63
It looks to me like there might be some possible collaboration opportunity
and overlap with the work you're doing here and the work @merrimanr is doing on
this PR - https://github.com/apache/metron/pull/1213#pullrequestreview-161248142
I'm just wondering if we might be able to kill 2 birds with one stone. We
probably don't want to change the MessageParser interface, but maybe we can
manage the bulk processing through a more generalized bridge between the
ParserBolt and parser implementations. I haven't dug too deep into
implementation feasibility, but it seems worth considering.
> Allow a grok statement to be applied to each line in a file.
> ------------------------------------------------------------
>
> Key: METRON-1761
> URL: https://issues.apache.org/jira/browse/METRON-1761
> Project: Metron
> Issue Type: Improvement
> Reporter: Laurens Vets
> Assignee: Otto Fowler
> Priority: Minor
>
> Make grok work where each line in incoming logs is a separate unit to be
> parsed.
> This would for instance allow NiFi to pick up log files (whereby each line is
> to be parsed separately) and send them to Metron without having to split the
> content.
> Example content of a log file where a grok statement needs to be applied to
> each line:
> {code:java}
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.000073 0.001048 0.000057 200 200 0 29 "GET http://www.example.com:80/
> HTTP/1.1" "curl/7.38.0" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.000086 0.001048 0.001337 200 200 0 57 "GET https://www.example.com:443/
> HTTP/1.1" "curl/7.38.0" DHE-RSA-AES128-SHA TLSv1.2
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001069 0.000028 0.000041 - - 82 305 "- - - " "-" - -
> 2015-05-13T23:39:43.945958Z my-loadbalancer 192.168.131.39:2817 10.0.0.1:80
> 0.001065 0.000015 0.000023 - - 57 502 "- - - " "-"
> ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
