Quick thought... Should USER_ACCOUNT be a string; 'USER_ACCOUNT'? Otherwise it is expecting that there is a field in your message named USER_ACCOUNT. If there is no such field, the value of the variable USER_ACCOUNT will be null. Try that out in the REPL and see if it explains what you are seeing.
On Mon, Dec 2, 2019 at 3:24 PM Nick Allen <[email protected]> wrote: > > On the alert screen, for each record, I can see the fields from the > enrichment as well. I've tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', > USER_ACCOUNT, 'cyber:verified-accounts', 'd') " but that throws a syntax > exception. Not sure what am I missing? > > You should debug issues like this using the Stellar REPL. Run the Stellar > REPL and try out the Stellar expressions that you mentioned. In most > cases, it should be fairly easy to determine what is wrong. My original > response included the steps to do this. > > If for some reason it is not clear after you have tried these expressions > in the REPL, respond back with what you are seeing in the REPL in contract > to what you are seeing in the streaming topology and I can try to help > further. > > > > > > On Mon, Dec 2, 2019 at 3:11 PM Sanket Sharma <[email protected]> > wrote: > >> Thanks Nick. Your help is greatly appreciated. Based on the feedback and >> documentation, I was able to setup a streaming enrichment: >> $METRON_HOME/config/zookeeper/parsers/VERIFIED_ACCOUNTS.json >> { >> "parserClassName" : "org.apache.metron.parsers.json.JSONMapParser", >> "writerClassName" : >> "org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter", >> "sensorTopic":"verified-accounts", >> "parserConfig": >> { >> "shew.table" : "cyber:verified-accounts", >> "shew.cf" : "d", >> "shew.keyColumns" : "USER_ACCOUNT", >> "shew.enrichmentType" : "VERIFIED_ACCOUNTS" >> } >> } >> Then, I configured a stellar enrichment as below. The idea was that an >> alert should be triggered if the account does not exist in the enrichment >> (enrichment is basically a whitelist of user accounts): >> $METRON_HOME/config/zookeeper/enrichment/VERIFIED_ACCOUNTS.json >> { >> "enrichment": { >> "fieldMap": { >> "stellar": { >> "config": { >> "is_alert": "ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', >> USER_ACCOUNT, 'cyber:verified-accounts', 'd') == false" >> } >> } >> }, >> "fieldToTypeMap": {}, >> "config": {} >> }, >> "threatIntel": { >> "fieldMap": {}, >> "fieldToTypeMap": {}, >> "config": {}, >> "triageConfig": { >> "riskLevelRules": [], >> "aggregator": "MAX", >> "aggregationConfig": {} >> } >> }, >> "configuration": {} >> } >> >> When I run the above sensors, it seems like ENRICHMENT_EXISTS actually >> joins the records on the given key instead of returning true or false. On >> the alert screen, for each record, I can see the fields from the enrichment >> as well. I've tried "not ENRICHMENT_EXISTS('VERIFIED_ACCOUNTS', >> USER_ACCOUNT, 'cyber:verified-accounts', 'd') " but that throws a syntax >> exception. >> Not sure what am I missing? >> >> You are spot on. That is exactly what we are looking for. I'll go ahead >> and open a JIRA to discuss it further. >> >> Thanks once again for your detailed feedback and responses. Much >> appreciated. >> >> Best regards, >> Sanket >> ------------------------------ >> *From:* Nick Allen <[email protected]> >> *Sent:* Monday, December 2, 2019 9:09 PM >> *To:* [email protected] <[email protected]> >> *Subject:* Re: Switching alert status >> >> > I was able to create the alerts as suggested. My understanding is that >> stellar rules (and scores) configured in the config UI will only be >> evaluated if the is_alert flag is set to true. Is that correct? >> >> Threat Triage is what gives you the "scores" that you are referring to. >> The purpose of Threat Triage is to assess a threat and output an overall >> threat score. That score can then be used to prioritize which threats need >> to be addressed first. >> >> Threat Triage only runs on messages where there is a field named >> "is_alert" with a Stellar expression that evaluates to true. This allows >> you to avoid the expense of Threat Triage, in cases where you know it is >> not needed. >> >> >> > Is there perhaps a way to toggle the is_alert flag based on the threat >> triage score ... or alternatively a way to calculate score without >> setting the is_alert flag to true? >> >> The "is_alert" field is used as a flag to indicate which messages should >> undergo Threat Triage processing. Once a message completes Threat Triage >> and gets a score, changing or removing the "is_alert" field does not do >> anything. >> >> >> > The use case is that we still want to score incoming records (based on >> stellar rules) but don’t want them displaying in the alerts UI unless they >> have a score... We could potentially filter the records from the alerts >> UI I suppose or clean up the Elastic index periodically, but wondering if >> something like this can be done out of the box? >> >> If you just want to filter messages from the Alerts UI, then you can add >> a search filter to filter out messages that do not have a score or have a >> really low score. A user can filter and sort messages however they like >> within the Alerts UI, but this may not be exactly what you want. >> >> If you want to *not* index messages (into Solr or Elasticsearch) unless >> the message has a score, then this is not directly possible >> out-of-the-box. Right now we index all messages into the search indices. >> >> It sounds like you might want to only index a subset of your messages. >> If you are so inclined, feel free to open a JIRA to discuss that as a new >> feature. >> >> >> Hope this helps >> >> >> >> >> >> >> >> >> On Wed, Nov 27, 2019 at 6:48 PM Sanket Sharma <[email protected]> >> wrote: >> >> Excellent! Thank you so much for the assistance. >> >> >> >> I was able to create the alerts as suggested. My understanding is that >> stellar rules (and scores) configured in the config UI will only be >> evaluated if the is_alert flag is set to true. Is that correct? >> >> >> >> Is there perhaps a way to toggle the is_alert flag based on the threat >> triage score (or alternatively a way to calculate score without setting the >> is_alert flag to true?) The use case is that we still want to score >> incoming records (based on stellar rules) but don’t want them displaying in >> the alerts UI unless they have a score. We could potentially filter the >> records from the alerts UI I suppose or clean up the Elastic index >> periodically, but wondering if something like this can done out of the box? >> >> >> >> Best regards, >> >> Sanket >> >> >> >> *From: *Nick Allen <[email protected]> >> *Reply to: *"[email protected]" <[email protected]> >> *Date: *Thursday, 21 November 2019 at 20:45 >> *To: *"[email protected]" <[email protected]> >> *Subject: *Re: Switching alert status >> >> >> >> Yes. You would create an enrichment using a Stellar expression that >> looks something like this. >> >> >> >> is_alert := IN_SUBNET(ip_src_addr, '192.168.0.0/24') >> >> >> >> I can create, test and load that enrichment using the Stellar REPL. >> >> >> >> [root@node1 0.7.2]# source /etc/default/metron >> >> [root@node1 0.7.2]# $METRON_HOME/bin/stellar -z $ZOOKEEPER >> >> >> >> Let's test out the expression to make sure it does what we would expect. >> You can make the expression as complex as you need for your use case using >> Stellar >> <https://metron.apache.org/current-book/metron-stellar/stellar-common/index.html> >> . >> >> >> >> [Stellar]>>> ip_src_addr := "192.168.0.22" >> 192.168.0.22 >> [Stellar]>>> is_alert := IN_SUBNET(ip_src_addr, '192.168.0.0/24') >> true >> >> Need more help? >> >> >> >> [Stellar]>>> ?IN_SUBNET >> IN_SUBNET >> Description: Returns true if an IP is within a subnet range. >> >> Arguments: >> ip - The IP address in string form >> cidr+ - One or more IP ranges specified in CIDR notation (for example >> 192.168.0.0/24) >> >> Returns: True if the IP address is within at least one of the network ranges >> and false if otherwise >> >> >> >> Then I can create and save the enrichment configuration necessary to >> apply your Stellar expression. >> >> >> >> [Stellar]>>> bro := SHELL_EDIT() >> >> { >> >> "enrichment" : { >> >> "fieldMap": { >> >> "stellar" : { >> >> "config" : { >> >> "is_alert" : "IN_SUBNET(ip_src_addr, '192.168.0.0/24')" >> >> } >> >> } >> >> } >> >> }, >> >> "threatIntel": { >> >> "fieldMap": {}, >> >> "fieldToTypeMap": {} >> >> } >> >> } >> >> [Stellar]>>> CONFIG_PUT("ENRICHMENT", bro, "bro") >> >> >> >> Hope this helps. Good luck. >> >> >> >> >> >> On Thu, Nov 21, 2019 at 1:21 PM Sanket Sharma <[email protected]> >> wrote: >> >> Hi, >> >> >> >> I was wondering if there is a way to switch the alert status on or off >> based on rule or condition? For instance, I have two different subnets >> where I’m monitoring network traffic. One is an open network and the other >> is a private/secured network. >> >> >> >> I would like to set ‘is_alert’ to true if the src is subnet two and set >> it to false when its subnet one. The end goal is to only have certain >> alerts displayed in the alerts UI based on conditions. Is it possible? >> >> >> >> >> >> Best regards, >> >> Sanket >> >>
