Thanks Nick. Your help is greatly appreciated. Based on the feedback and
documentation, I was able to setup a streaming enrichment:
$METRON_HOME/config/zookeeper/parsers/VERIFIED_ACCOUNTS.json
{
"parserClassName" : "org.apache.metron.parsers.json.JSONMapParser",
"writerClassName" :
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
"sensorTopic":"verified-accounts",
"parserConfig":
{
"shew.table" : "cyber:verified-accounts",
"shew.cf" : "d",
"shew.keyColumns" : "USER_ACCOUNT",
"shew.enrichmentType" : "VERIFIED_ACCOUNTS"
}
}
Then, I configured a stellar enrichment as below. The idea was that an alert
should be triggered if the account does not exist in the enrichment (enrichment
is basically a whitelist of user accounts):
$METRON_HOME/config/zookeeper/enrichment/VERIFIED_ACCOUNTS.json
{
"enrichment": {
"fieldMap": {
"stellar": {
"config": {
"is_alert": "ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
USER_ACCOUNT, 'cyber:verified-accounts', 'd') == false"
}
}
},
"fieldToTypeMap": {},
"config": {}
},
"threatIntel": {
"fieldMap": {},
"fieldToTypeMap": {},
"config": {},
"triageConfig": {
"riskLevelRules": [],
"aggregator": "MAX",
"aggregationConfig": {}
}
},
"configuration": {}
}
When I run the above sensors, it seems like ENRICHMENT_EXISTS actually joins
the records on the given key instead of returning true or false. On the alert
screen, for each record, I can see the fields from the enrichment as well. I've
tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', USER_ACCOUNT,
'cyber:verified-accounts', 'd') " but that throws a syntax exception.
Not sure what am I missing?
You are spot on. That is exactly what we are looking for. I'll go ahead and
open a JIRA to discuss it further.
Thanks once again for your detailed feedback and responses. Much appreciated.
Best regards,
Sanket
________________________________
From: Nick Allen <[email protected]>
Sent: Monday, December 2, 2019 9:09 PM
To: [email protected] <[email protected]>
Subject: Re: Switching alert status
> I was able to create the alerts as suggested. My understanding is that
> stellar rules (and scores) configured in the config UI will only be evaluated
> if the is_alert flag is set to true. Is that correct?
Threat Triage is what gives you the "scores" that you are referring to. The
purpose of Threat Triage is to assess a threat and output an overall threat
score. That score can then be used to prioritize which threats need to be
addressed first.
Threat Triage only runs on messages where there is a field named "is_alert"
with a Stellar expression that evaluates to true. This allows you to avoid the
expense of Threat Triage, in cases where you know it is not needed.
> Is there perhaps a way to toggle the is_alert flag based on the threat triage
> score ... or alternatively a way to calculate score without setting the
> is_alert flag to true?
The "is_alert" field is used as a flag to indicate which messages should
undergo Threat Triage processing. Once a message completes Threat Triage and
gets a score, changing or removing the "is_alert" field does not do anything.
> The use case is that we still want to score incoming records (based on
> stellar rules) but don’t want them displaying in the alerts UI unless they
> have a score... We could potentially filter the records from the alerts UI I
> suppose or clean up the Elastic index periodically, but wondering if
> something like this can be done out of the box?
If you just want to filter messages from the Alerts UI, then you can add a
search filter to filter out messages that do not have a score or have a really
low score. A user can filter and sort messages however they like within the
Alerts UI, but this may not be exactly what you want.
If you want to not index messages (into Solr or Elasticsearch) unless the
message has a score, then this is not directly possible out-of-the-box. Right
now we index all messages into the search indices.
It sounds like you might want to only index a subset of your messages. If you
are so inclined, feel free to open a JIRA to discuss that as a new feature.
Hope this helps
On Wed, Nov 27, 2019 at 6:48 PM Sanket Sharma
<[email protected]<mailto:[email protected]>> wrote:
Excellent! Thank you so much for the assistance.
I was able to create the alerts as suggested. My understanding is that stellar
rules (and scores) configured in the config UI will only be evaluated if the
is_alert flag is set to true. Is that correct?
Is there perhaps a way to toggle the is_alert flag based on the threat triage
score (or alternatively a way to calculate score without setting the is_alert
flag to true?) The use case is that we still want to score incoming records
(based on stellar rules) but don’t want them displaying in the alerts UI unless
they have a score. We could potentially filter the records from the alerts UI
I suppose or clean up the Elastic index periodically, but wondering if
something like this can done out of the box?
Best regards,
Sanket
From: Nick Allen <[email protected]<mailto:[email protected]>>
Reply to: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, 21 November 2019 at 20:45
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Switching alert status
Yes. You would create an enrichment using a Stellar expression that looks
something like this.
is_alert := IN_SUBNET(ip_src_addr, '192.168.0.0/24<http://192.168.0.0/24>')
I can create, test and load that enrichment using the Stellar REPL.
[root@node1 0.7.2]# source /etc/default/metron
[root@node1 0.7.2]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
Let's test out the expression to make sure it does what we would expect. You
can make the expression as complex as you need for your use case using
Stellar<https://metron.apache.org/current-book/metron-stellar/stellar-common/index.html>.
[Stellar]>>> ip_src_addr := "192.168.0.22"
192.168.0.22
[Stellar]>>> is_alert := IN_SUBNET(ip_src_addr,
'192.168.0.0/24<http://192.168.0.0/24>')
true
Need more help?
[Stellar]>>> ?IN_SUBNET
IN_SUBNET
Description: Returns true if an IP is within a subnet range.
Arguments:
ip - The IP address in string form
cidr+ - One or more IP ranges specified in CIDR notation (for example
192.168.0.0/24<http://192.168.0.0/24>)
Returns: True if the IP address is within at least one of the network ranges
and false if otherwise
Then I can create and save the enrichment configuration necessary to apply your
Stellar expression.
[Stellar]>>> bro := SHELL_EDIT()
{
"enrichment" : {
"fieldMap": {
"stellar" : {
"config" : {
"is_alert" : "IN_SUBNET(ip_src_addr,
'192.168.0.0/24'<http://192.168.0.0/24'>)"
}
}
}
},
"threatIntel": {
"fieldMap": {},
"fieldToTypeMap": {}
}
}
[Stellar]>>> CONFIG_PUT("ENRICHMENT", bro, "bro")
Hope this helps. Good luck.
On Thu, Nov 21, 2019 at 1:21 PM Sanket Sharma
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I was wondering if there is a way to switch the alert status on or off based on
rule or condition? For instance, I have two different subnets where I’m
monitoring network traffic. One is an open network and the other is a
private/secured network.
I would like to set ‘is_alert’ to true if the src is subnet two and set it to
false when its subnet one. The end goal is to only have certain alerts
displayed in the alerts UI based on conditions. Is it possible?
Best regards,
Sanket
