> On the alert screen, for each record, I can see the fields from the
enrichment as well. I've tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS',
USER_ACCOUNT, 'cyber:verified-accounts', 'd') " but that throws a syntax
exception. Not sure what am I missing?You should debug issues like this using the Stellar REPL. Run the Stellar REPL and try out the Stellar expressions that you mentioned. In most cases, it should be fairly easy to determine what is wrong. My original response included the steps to do this. If for some reason it is not clear after you have tried these expressions in the REPL, respond back with what you are seeing in the REPL in contract to what you are seeing in the streaming topology and I can try to help further. On Mon, Dec 2, 2019 at 3:11 PM Sanket Sharma <[email protected]> wrote: > Thanks Nick. Your help is greatly appreciated. Based on the feedback and > documentation, I was able to setup a streaming enrichment: > $METRON_HOME/config/zookeeper/parsers/VERIFIED_ACCOUNTS.json > { > "parserClassName" : "org.apache.metron.parsers.json.JSONMapParser", > "writerClassName" : > "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter", > "sensorTopic":"verified-accounts", > "parserConfig": > { > "shew.table" : "cyber:verified-accounts", > "shew.cf" : "d", > "shew.keyColumns" : "USER_ACCOUNT", > "shew.enrichmentType" : "VERIFIED_ACCOUNTS" > } > } > Then, I configured a stellar enrichment as below. The idea was that an > alert should be triggered if the account does not exist in the enrichment > (enrichment is basically a whitelist of user accounts): > $METRON_HOME/config/zookeeper/enrichment/VERIFIED_ACCOUNTS.json > { > "enrichment": { > "fieldMap": { > "stellar": { > "config": { > "is_alert": "ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', > USER_ACCOUNT, 'cyber:verified-accounts', 'd') == false" > } > } > }, > "fieldToTypeMap": {}, > "config": {} > }, > "threatIntel": { > "fieldMap": {}, > "fieldToTypeMap": {}, > "config": {}, > "triageConfig": { > "riskLevelRules": [], > "aggregator": "MAX", > "aggregationConfig": {} > } > }, > "configuration": {} > } > > When I run the above sensors, it seems like ENRICHMENT_EXISTS actually > joins the records on the given key instead of returning true or false. On > the alert screen, for each record, I can see the fields from the enrichment > as well. I've tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', > USER_ACCOUNT, 'cyber:verified-accounts', 'd') " but that throws a syntax > exception. > Not sure what am I missing? > > You are spot on. That is exactly what we are looking for. I'll go ahead > and open a JIRA to discuss it further. > > Thanks once again for your detailed feedback and responses. Much > appreciated. > > Best regards, > Sanket > ------------------------------ > *From:* Nick Allen <[email protected]> > *Sent:* Monday, December 2, 2019 9:09 PM > *To:* [email protected] <[email protected]> > *Subject:* Re: Switching alert status > > > I was able to create the alerts as suggested. My understanding is that > stellar rules (and scores) configured in the config UI will only be > evaluated if the is_alert flag is set to true. Is that correct? > > Threat Triage is what gives you the "scores" that you are referring to. > The purpose of Threat Triage is to assess a threat and output an overall > threat score. That score can then be used to prioritize which threats need > to be addressed first. > > Threat Triage only runs on messages where there is a field named > "is_alert" with a Stellar expression that evaluates to true. This allows > you to avoid the expense of Threat Triage, in cases where you know it is > not needed. > > > > Is there perhaps a way to toggle the is_alert flag based on the threat > triage score ... or alternatively a way to calculate score without > setting the is_alert flag to true? > > The "is_alert" field is used as a flag to indicate which messages should > undergo Threat Triage processing. Once a message completes Threat Triage > and gets a score, changing or removing the "is_alert" field does not do > anything. > > > > The use case is that we still want to score incoming records (based on > stellar rules) but don’t want them displaying in the alerts UI unless they > have a score... We could potentially filter the records from the alerts > UI I suppose or clean up the Elastic index periodically, but wondering if > something like this can be done out of the box? > > If you just want to filter messages from the Alerts UI, then you can add > a search filter to filter out messages that do not have a score or have a > really low score. A user can filter and sort messages however they like > within the Alerts UI, but this may not be exactly what you want. > > If you want to *not* index messages (into Solr or Elasticsearch) unless > the message has a score, then this is not directly possible > out-of-the-box. Right now we index all messages into the search indices. > > It sounds like you might want to only index a subset of your messages. If > you are so inclined, feel free to open a JIRA to discuss that as a new > feature. > > > Hope this helps > > > > > > > > > On Wed, Nov 27, 2019 at 6:48 PM Sanket Sharma <[email protected]> > wrote: > > Excellent! Thank you so much for the assistance. > > > > I was able to create the alerts as suggested. My understanding is that > stellar rules (and scores) configured in the config UI will only be > evaluated if the is_alert flag is set to true. Is that correct? > > > > Is there perhaps a way to toggle the is_alert flag based on the threat > triage score (or alternatively a way to calculate score without setting the > is_alert flag to true?) The use case is that we still want to score > incoming records (based on stellar rules) but don’t want them displaying in > the alerts UI unless they have a score. We could potentially filter the > records from the alerts UI I suppose or clean up the Elastic index > periodically, but wondering if something like this can done out of the box? > > > > Best regards, > > Sanket > > > > *From: *Nick Allen <[email protected]> > *Reply to: *"[email protected]" <[email protected]> > *Date: *Thursday, 21 November 2019 at 20:45 > *To: *"[email protected]" <[email protected]> > *Subject: *Re: Switching alert status > > > > Yes. You would create an enrichment using a Stellar expression that looks > something like this. > > > > is_alert := IN_SUBNET(ip_src_addr, '192.168.0.0/24') > > > > I can create, test and load that enrichment using the Stellar REPL. > > > > [root@node1 0.7.2]# source /etc/default/metron > > [root@node1 0.7.2]# $METRON_HOME/bin/stellar -z $ZOOKEEPER > > > > Let's test out the expression to make sure it does what we would expect. > You can make the expression as complex as you need for your use case using > Stellar > <https://metron.apache.org/current-book/metron-stellar/stellar-common/index.html> > . > > > > [Stellar]>>> ip_src_addr := "192.168.0.22" > 192.168.0.22 > [Stellar]>>> is_alert := IN_SUBNET(ip_src_addr, '192.168.0.0/24') > true > > Need more help? > > > > [Stellar]>>> ?IN_SUBNET > IN_SUBNET > Description: Returns true if an IP is within a subnet range. > > Arguments: > ip - The IP address in string form > cidr+ - One or more IP ranges specified in CIDR notation (for example > 192.168.0.0/24) > > Returns: True if the IP address is within at least one of the network ranges > and false if otherwise > > > > Then I can create and save the enrichment configuration necessary to apply > your Stellar expression. > > > > [Stellar]>>> bro := SHELL_EDIT() > > { > > "enrichment" : { > > "fieldMap": { > > "stellar" : { > > "config" : { > > "is_alert" : "IN_SUBNET(ip_src_addr, '192.168.0.0/24')" > > } > > } > > } > > }, > > "threatIntel": { > > "fieldMap": {}, > > "fieldToTypeMap": {} > > } > > } > > [Stellar]>>> CONFIG_PUT("ENRICHMENT", bro, "bro") > > > > Hope this helps. Good luck. > > > > > > On Thu, Nov 21, 2019 at 1:21 PM Sanket Sharma <[email protected]> > wrote: > > Hi, > > > > I was wondering if there is a way to switch the alert status on or off > based on rule or condition? For instance, I have two different subnets > where I’m monitoring network traffic. One is an open network and the other > is a private/secured network. > > > > I would like to set ‘is_alert’ to true if the src is subnet two and set it > to false when its subnet one. The end goal is to only have certain alerts > displayed in the alerts UI based on conditions. Is it possible? > > > > > > Best regards, > > Sanket > >
