[
https://issues.apache.org/jira/browse/METRON-1657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546869#comment-16546869
]
ASF GitHub Bot commented on METRON-1657:
----------------------------------------
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/1099#discussion_r203095632
--- Diff: use-cases/parser_chaining/README.md ---
@@ -233,3 +233,10 @@ cat ~/data.log |
/usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --b
```
You should see indices created for the `cisco-5-304` and `cisco-6-302`
data with appropriate fields created for each type.
+
+# Aggregated Parsers with Parser Chaining
+Chained parsers can be run as aggregated parsers. These parsers continue
to use the sensor specific Kafka topics, and do not do internal routing to the
appropriate sensor.
+
--- End diff --
I think adding the follow on is a great idea. I wonder if we shouldn't
change the default install to aggregate the default sensors? I don't think
that will work though, because the comma separation in ambari is the list, and
you won't be able to do it.
My main concern is that the reviewers and committers of this pr are going
to be the only ones who can do it, and everyone in user land is going to be
lost. If this is going to be expert only ( until the UI comes ) and not
recommend, or a preview thing, maybe mark it as such.
> Parser aggregation in storm
> ---------------------------
>
> Key: METRON-1657
> URL: https://issues.apache.org/jira/browse/METRON-1657
> Project: Metron
> Issue Type: Bug
> Reporter: Justin Leet
> Assignee: Justin Leet
> Priority: Major
>
> Currently our parsing solution requires one storm topology per sensor. It has
> been complained that this may be wasteful of resources and that, rather than
> one storm topology per sensor, it would be advantageous to have multiple
> sensors in the same topology. The benefit to this is that it would require
> fewer storm slots.
> The issue with this is that whenever we've aggregated functionality like this
> before, we've run into issues appropriately being able to scale storm (e.g.
> batch vs random access indexing in the same topology). The main point in
> addressing this is to recommend that parsers with similar velocities and
> complexity are grouped together.
> Particularly for a first cut, leave the configuration mostly as-is, while
> allowing for comma separated lists of sensors in start_parser_topology.sh
> (e.g. bro,yaf creates a aggregated parser consisting of those two).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
