Here's the full work log in one place:
1. Enrichment configuration:
"enrichment": {
"fieldMap": {
"stellar": {
"config": {
"is_alert": "not(ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
USER_ACCOUNT , 'cyber:verified-accounts', 'd'))"
}
}
},
"fieldToTypeMap": {},
"config": {}
}
USER_ACCOUNT is a field in the incoming JSON message. We try and look it up in
the enrichment table 'cyber:verified-accounts'. We only need to raise the alert
if that user id is not present in the HBase table.
2. Stellar Tests (Have redacted the full account ids):
[Stellar]>>> USER_ACCOUNT := "xxxxxxxxxxx43254"
xxxxxxxxxxx43254
[Stellar]>>> is_alert := not(ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
USER_ACCOUNT, 'cyber:verified-accounts', 'd'))
false
[Stellar]>>> USER_ACCOUNT := "xxxxxxxxxxx70293"
xxxxxxxxxxx70293
[Stellar]>>> is_alert := not(ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
USER_ACCOUNT, 'cyber:verified-accounts', 'd'))
false
[Stellar]>>>
3. Incoming JSON message:
{
...........................,
...........................,
...........................,
"USER_ACCOUNT": "xxxxxxxxxxx43254",
...........................,
...........................,
...........................,
...........................,
...........................,
...........................,
...........................,
...........................,
...........................,
}
In the alerts UI is_alert is set to true for both of them and I can see other
fields from the verified-accounts HBase table:
[cid:f331790b-8f00-4a53-a199-30a307accc9d]
So it seems that Stellar REPL is working as expected but the Enrichments from
configuration are returning true for everything. I am trying to restart the
topology and see if that makes a difference.
Thank you very much for your help and prompt responses.
Best regards,
Sanket
________________________________
From: Nick Allen <[email protected]>
Sent: Monday, December 2, 2019 10:28 PM
To: Sanket Sharma <[email protected]>
Cc: [email protected] <[email protected]>
Subject: Re: Switching alert status
Quick thought... Should USER_ACCOUNT be a string; 'USER_ACCOUNT'? Otherwise it
is expecting that there is a field in your message named USER_ACCOUNT. If
there is no such field, the value of the variable USER_ACCOUNT will be null.
Try that out in the REPL and see if it explains what you are seeing.
On Mon, Dec 2, 2019 at 3:24 PM Nick Allen
<[email protected]<mailto:[email protected]>> wrote:
> On the alert screen, for each record, I can see the fields from the
> enrichment as well. I've tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
> USER_ACCOUNT, 'cyber:verified-accounts', 'd') " but that throws a syntax
> exception. Not sure what am I missing?
You should debug issues like this using the Stellar REPL. Run the Stellar REPL
and try out the Stellar expressions that you mentioned. In most cases, it
should be fairly easy to determine what is wrong. My original response included
the steps to do this.
If for some reason it is not clear after you have tried these expressions in
the REPL, respond back with what you are seeing in the REPL in contract to what
you are seeing in the streaming topology and I can try to help further.
On Mon, Dec 2, 2019 at 3:11 PM Sanket Sharma
<[email protected]<mailto:[email protected]>> wrote:
Thanks Nick. Your help is greatly appreciated. Based on the feedback and
documentation, I was able to setup a streaming enrichment:
$METRON_HOME/config/zookeeper/parsers/VERIFIED_ACCOUNTS.json
{
"parserClassName" : "org.apache.metron.parsers.json.JSONMapParser",
"writerClassName" :
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
"sensorTopic":"verified-accounts",
"parserConfig":
{
"shew.table" : "cyber:verified-accounts",
"shew.cf<http://shew.cf>" : "d",
"shew.keyColumns" : "USER_ACCOUNT",
"shew.enrichmentType" : "VERIFIED_ACCOUNTS"
}
}
Then, I configured a stellar enrichment as below. The idea was that an alert
should be triggered if the account does not exist in the enrichment (enrichment
is basically a whitelist of user accounts):
$METRON_HOME/config/zookeeper/enrichment/VERIFIED_ACCOUNTS.json
{
"enrichment": {
"fieldMap": {
"stellar": {
"config": {
"is_alert": "ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
USER_ACCOUNT, 'cyber:verified-accounts', 'd') == false"
}
}
},
"fieldToTypeMap": {},
"config": {}
},
"threatIntel": {
"fieldMap": {},
"fieldToTypeMap": {},
"config": {},
"triageConfig": {
"riskLevelRules": [],
"aggregator": "MAX",
"aggregationConfig": {}
}
},
"configuration": {}
}
When I run the above sensors, it seems like ENRICHMENT_EXISTS actually joins
the records on the given key instead of returning true or false. On the alert
screen, for each record, I can see the fields from the enrichment as well. I've
tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', USER_ACCOUNT,
'cyber:verified-accounts', 'd') " but that throws a syntax exception.
Not sure what am I missing?
You are spot on. That is exactly what we are looking for. I'll go ahead and
open a JIRA to discuss it further.
Thanks once again for your detailed feedback and responses. Much appreciated.
Best regards,
Sanket
________________________________
From: Nick Allen <[email protected]<mailto:[email protected]>>
Sent: Monday, December 2, 2019 9:09 PM
To: [email protected]<mailto:[email protected]>
<[email protected]<mailto:[email protected]>>
Subject: Re: Switching alert status
> I was able to create the alerts as suggested. My understanding is that
> stellar rules (and scores) configured in the config UI will only be evaluated
> if the is_alert flag is set to true. Is that correct?
Threat Triage is what gives you the "scores" that you are referring to. The
purpose of Threat Triage is to assess a threat and output an overall threat
score. That score can then be used to prioritize which threats need to be
addressed first.
Threat Triage only runs on messages where there is a field named "is_alert"
with a Stellar expression that evaluates to true. This allows you to avoid the
expense of Threat Triage, in cases where you know it is not needed.
> Is there perhaps a way to toggle the is_alert flag based on the threat triage
> score ... or alternatively a way to calculate score without setting the
> is_alert flag to true?
The "is_alert" field is used as a flag to indicate which messages should
undergo Threat Triage processing. Once a message completes Threat Triage and
gets a score, changing or removing the "is_alert" field does not do anything.
> The use case is that we still want to score incoming records (based on
> stellar rules) but don’t want them displaying in the alerts UI unless they
> have a score... We could potentially filter the records from the alerts UI I
> suppose or clean up the Elastic index periodically, but wondering if
> something like this can be done out of the box?
If you just want to filter messages from the Alerts UI, then you can add a
search filter to filter out messages that do not have a score or have a really
low score. A user can filter and sort messages however they like within the
Alerts UI, but this may not be exactly what you want.
If you want to not index messages (into Solr or Elasticsearch) unless the
message has a score, then this is not directly possible out-of-the-box. Right
now we index all messages into the search indices.
It sounds like you might want to only index a subset of your messages. If you
are so inclined, feel free to open a JIRA to discuss that as a new feature.
Hope this helps
On Wed, Nov 27, 2019 at 6:48 PM Sanket Sharma
<[email protected]<mailto:[email protected]>> wrote:
Excellent! Thank you so much for the assistance.
I was able to create the alerts as suggested. My understanding is that stellar
rules (and scores) configured in the config UI will only be evaluated if the
is_alert flag is set to true. Is that correct?
Is there perhaps a way to toggle the is_alert flag based on the threat triage
score (or alternatively a way to calculate score without setting the is_alert
flag to true?) The use case is that we still want to score incoming records
(based on stellar rules) but don’t want them displaying in the alerts UI unless
they have a score. We could potentially filter the records from the alerts UI
I suppose or clean up the Elastic index periodically, but wondering if
something like this can done out of the box?
Best regards,
Sanket
From: Nick Allen <[email protected]<mailto:[email protected]>>
Reply to: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Thursday, 21 November 2019 at 20:45
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Re: Switching alert status
Yes. You would create an enrichment using a Stellar expression that looks
something like this.
is_alert := IN_SUBNET(ip_src_addr, '192.168.0.0/24<http://192.168.0.0/24>')
I can create, test and load that enrichment using the Stellar REPL.
[root@node1 0.7.2]# source /etc/default/metron
[root@node1 0.7.2]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
Let's test out the expression to make sure it does what we would expect. You
can make the expression as complex as you need for your use case using
Stellar<https://metron.apache.org/current-book/metron-stellar/stellar-common/index.html>.
[Stellar]>>> ip_src_addr := "192.168.0.22"
192.168.0.22
[Stellar]>>> is_alert := IN_SUBNET(ip_src_addr,
'192.168.0.0/24<http://192.168.0.0/24>')
true
Need more help?
[Stellar]>>> ?IN_SUBNET
IN_SUBNET
Description: Returns true if an IP is within a subnet range.
Arguments:
ip - The IP address in string form
cidr+ - One or more IP ranges specified in CIDR notation (for example
192.168.0.0/24<http://192.168.0.0/24>)
Returns: True if the IP address is within at least one of the network ranges
and false if otherwise
Then I can create and save the enrichment configuration necessary to apply your
Stellar expression.
[Stellar]>>> bro := SHELL_EDIT()
{
"enrichment" : {
"fieldMap": {
"stellar" : {
"config" : {
"is_alert" : "IN_SUBNET(ip_src_addr,
'192.168.0.0/24'<http://192.168.0.0/24'>)"
}
}
}
},
"threatIntel": {
"fieldMap": {},
"fieldToTypeMap": {}
}
}
[Stellar]>>> CONFIG_PUT("ENRICHMENT", bro, "bro")
Hope this helps. Good luck.
On Thu, Nov 21, 2019 at 1:21 PM Sanket Sharma
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I was wondering if there is a way to switch the alert status on or off based on
rule or condition? For instance, I have two different subnets where I’m
monitoring network traffic. One is an open network and the other is a
private/secured network.
I would like to set ‘is_alert’ to true if the src is subnet two and set it to
false when its subnet one. The end goal is to only have certain alerts
displayed in the alerts UI based on conditions. Is it possible?
Best regards,
Sanket
