[
https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547023#comment-16547023
]
ASF GitHub Bot commented on METRON-1657:
----------------------------------------
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/1099
A mechanism for the routing process to apply a transform or some such.
@cestella may have a better design idea.
What I would like us to do is remove the transport from the message where
there are common wrappers.
Example: All the message types delivered by syslog. The parsers should
not have to all parser syslog AND the message. I imagine defining a
transform/parser in the router that takes every message and transforms it (
parses syslog fields + structured data if 5424 into meta and MSG to the bytes )
and then passes it on to a parser that only needs to know the message.
That way we can have simpler parsers, and even support the same message
when transported in different wrappers.
We can talk about if this is a function of parser chaining, such that each
specialized parser will be second in a chain with the syslog unwrapped being
first, or part of routing, but I think it is part of routing personally.
> Parser aggregation in storm
> ---------------------------
>
> Key: METRON-1657
> URL: https://issues.apache.org/jira/browse/METRON-1657
> Project: Metron
> Issue Type: Bug
> Reporter: Justin Leet
> Assignee: Justin Leet
> Priority: Major
>
> Currently our parsing solution requires one storm topology per sensor. It has
> been complained that this may be wasteful of resources and that, rather than
> one storm topology per sensor, it would be advantageous to have multiple
> sensors in the same topology. The benefit to this is that it would require
> fewer storm slots.
> The issue with this is that whenever we've aggregated functionality like this
> before, we've run into issues appropriately being able to scale storm (e.g.
> batch vs random access indexing in the same topology). The main point in
> addressing this is to recommend that parsers with similar velocities and
> complexity are grouped together.
> Particularly for a first cut, leave the configuration mostly as-is, while
> allowing for comma separated lists of sensors in start_parser_topology.sh
> (e.g. bro,yaf creates a aggregated parser consisting of those two).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
