Hi,
Thank you for starting a great discussion! We started exploring Metron in June
this for networking monitoring. We are piloting it with an objective of
replacing Splunk in certain or perhaps all scenarios. We’re looking at about 2
TB of data per day.
1. Features we are currently considering:
* Enrichments
* Streaming enhancements: We are using Spark to do some enrichments but
need to explore this further.
* Profiler: Not using it at the moment
* Pcap: Not using it at the moment.
* Flatfile summarizer: Not using it at the moment.
* MaaS: IMHO this needs serious usability enhancements, especially for
data scientists. Deploying models seems like a common issue that most data
scientist struggle with (at least in our area, unless they have serious
python/engineering skills.).
* Meta alerts: Not using it at the moment
* Parser aggregation: Limited use
* Config UI: Using it extensively to configure sensors and rules.
* Alert UI: Using it extensive to view alerts.
* Elastic search: Using it extensively to index alerts and other data.
* Stellar: Not using it at the moment, except for creating rules with
scores in the config UI.
* Stellar REPL: Not using it at all
* REST API: Not using it explicitly.
* Other?
2. Many features around usability can be improved:
* Model deployment can reconsidered as a whole.
* Ability to compare models
* Config UI field configuration could be improved
* General ease of use/deployment, documentation
* Templates for common use cases
* Reports – we just can’t do without reporting in the enterprise ☺
3. Alerts UI, Stellar and pipelines I suppose.
4. I would love to contribute ☺, just in the middle of a big relocation.
Hopefully, I will be able to resume and join the community in next 2-3 months.
We have another interesting use case where we kind of started prototyping
Metron – financial fraud. Although it might sound a very different and
unrelated domain, the “technical architectural pattern” is astonishingly
similar. We receive streaming and batch data from various channels over kafka,
gets enriched and the based on certain rules we assign a score to it. It then
makes it to the alert UI where investigators can further examine the
transactions. This is obviously an oversimplification, but I hope you get the
idea.
I was thinking of proposing a fork or perhaps a different “flavour” of metron
that caters for finance domain and can be built as a separate project, although
not sure how to go about it. Is that something the community/project owners
might be interested in considering or supporting?
Best regards,
Sanket
From: Michael Miklavcic <[email protected]>
Reply to: "[email protected]" <[email protected]>
Date: Thursday, 17 October 2019 at 18:22
To: "[email protected]" <[email protected]>, "[email protected]"
<[email protected]>
Subject: [DISCUSS] How are you using in Metron?
I'd like to kick off a discussion to get a sense of how the broader community
is currently using Metron.
1. What features are you using or seriously considering? e.g.
1. enrichments
2. streaming enrichments
3. profiler
4. pcap
5. flatfile summarizer
6. MaaS
7. Meta alerts
8. parser aggregation
9. config UI
10. alert UI
11. solr, ES
12. Stellar
13. Stellar REPL
14. REST API
15. other?
2. What features would you like to see added or improved?
3. What features do you consider to be core to Metron as a platform?
4. If you're using Metron, but not an active community contributor, what
would it take to get you more involved in the project?
We are close to finishing up a feature branch around upgrading to HDP 3.1, and
subsequently on the doorstep of a 1.0 release. This is a huge milestone for the
project. I think it's time to take some lessons learned over the past several
years and consider what the next phase of Metron will be. Whether you've
participated in community discussions before or not, we'd love to hear from you.
Best,
Mike Miklavcic
PMC Apache Metron
