To an extent it very much depends on the use case. I have seen over a million
EPS on a six node cluster for pcap and basic net flow. If you add a lot of
complex enrichment and profiling that will obviously increase the load. Tuning
the components for the workload can also make a significant diff
Is there a rough guide to match EPS to an architectural sizing guide? I know
its very difficult to extrapolate out, but a rough estimate would be nice. This
may have already been attempted, and if yes, then please disregard.
Or can anyone share what they have found to work best?
For example,
is there any documentation to create new sensor in metron?
On Wed, 18 Oct 2017 at 01.22 Simon Elliston Ball <
si...@simonellistonball.com> wrote:
> Best bet there is to create a new sensor config using the grok parser
> type. So you would for example have a kafka topic called host_dhcp and a
> se
Best bet there is to create a new sensor config using the grok parser type. So
you would for example have a kafka topic called host_dhcp and a sensor called
host_dhcp with the relevant grok pattern.
Simon
> On 17 Oct 2017, at 19:19, Youzha wrote:
>
> that’s what i mean.
> what sensor that i
that’s what i mean.
what sensor that i need if i want to do this case?
especially when i wanna parse some host logs into metron enrichment and
indexing
On Wed, 18 Oct 2017 at 01.03 Simon Elliston Ball <
si...@simonellistonball.com> wrote:
> What you want to do in this setting is just TailFile, th
What you want to do in this setting is just TailFile, the just push to Kafka.
The grok piece is more efficiently handled in the Metron grok parser.
Push to a kafka topic named for your sensor, then setup a sensor (a parser
topology to do the grok parsing and any transformation you need). Each se
after nifi procces :
TAILFILE -> TRANSFORM_TO_GROK -> PUSH_KAFKA
what metron topology that i can use to procces the data in kafka? so it can
be enrichment by metron. i’ve check the article about adding new telemetry
source with squid, there is a squid topology that will ingest from the
squid topi
is there a guide of sorts we can follow, or noodle through, to write our own
java based parser?
or do we need to just java through and figure it out?
From: Otto Fowler
Sent: Tuesday, October 17, 2017 1:30 PM
To: Youzha; user@metron.apache.org
Subject: Re: even
So,
There are several options parsing the data and enriching.
1. A native parser ( java ), which you have noticed is not there
2. An instance of the GROK parser, with GROK rules that parser the input
3. If it is CSV an instance of the CSV parser
4. If it is JSON an instance of the JSONMap pars
Hi Lauren thx for your reply,
yeah your suggestion absolutely right. i was able to ingest the logs to
kafka. but how metron can enrich and index all of it? i think there are
only bro, snort, yaf, snort, pcap, websphere topology storm on metron for
parsers. so, how metron can read the logs telemet
I am so noob in all of this. I am using full-dev vm metron install to do my
research. So I have 2 options to install snort: as per my understanding
1- Install it in a usual way (like that on a regular linux machine) and
then make its kafka topic
2- Use ansible role to do all of that. Read the con
No special commands. Install and configure Snort however you like and get
those logs into a Kafka topic. Metron is completely agnostic to how sensor
telemetry lands in Kafka.
We also have an Ansible role that will install Snort along with a simple
mechanism to transport its logs to Kafka. This
Hi Youzha,
Either check how the snort logs on the full dev installation are
ingested (I believe it's with a script) or check the Apache NiFi project
which makes it very easy to read logs from almost any format and ingest
them to Metron via Kafka.
On 2017-10-17 08:53, Youzha wrote:
> is it poss
Ok, Now I get it. Now should I install snort in vagrant ssh in the normal
way snort is usually install on a linux distro or do I need to run some
special commands again?
On Tue, Oct 17, 2017 at 7:45 PM, Nick Allen wrote:
> In the Full Dev environment, Snort is not installed. We install "Sensor
-- Forwarded message -
From: Youzha
Date: Tue, 17 Oct 2017 at 22.53
Subject: Re: event correlation on metron
To:
is it possible to ingest other logs like /var/log/secure for example to be
new telemetry on metron? i’ve seen the metron architecture on the website
like picture belo
is it possible to ingest other logs like /var/log/secure for example to be
new telemetry on metron? i’ve seen the metron architecture on the website
like picture below. host logs, email, av, etc can be telemetry event buffer
on metron. if this possible, could you give me some suggestion how to do i
In the Full Dev environment, Snort is not installed. We install "Sensor
Stubs" which is just a mechanism that continually replays canned telemetry
logs repetitively to mimic real sensors. We have to do this because of
resource constraints when running all of Metron on a single VM. See the
follow
yes,, but when i do snort -v in vagrant ssh console it says snort isnt
installed where as it can be seen working in metron. Due to that reason I
am confused because James Sirota said to install snort.
On Tue, Oct 17, 2017 at 7:05 PM, Nick Allen wrote:
> From Metron's perspective, Snort is just a
>From Metron's perspective, Snort is just another sensor. Snort is
installed, managed and executed completely independent of Metron itself. As
with any sensor, you are responsible for getting the telemetry produced by
Snort into Kafka. Metron can then consume that telemetry from Kafka and do
wond
If you want to look at failed login attempts for each user over time, then
the Profiler might be a good solution. Your profile will depend on the
fields available in your telemetry, but it would look something like this,
as an example.
{
"profile": "failed-logins",
"foreach": "user.name",
"
for example,
i wanna try to correlate between logs.
how many times user A have login failed and how many times user A have
login succeed. include detail IP, timestamp etc.
is this possible to do with metron?
On 17/10/17 02:56, James Sirota wrote:
What specifically are you looking to correla
And I am sorry about one confusion but isnt snort builtin into the metron
framework? If so then cant we access that snort and do the tasks you
mentioned earlier?
On Tue, Oct 17, 2017 at 11:39 AM, Syed Hammad Tahir
wrote:
> Hi,
>
> Thanks for the support. Can it be performed both on dumped log an
22 matches
Mail list logo
