Hi Nick, Thanks for the explanation. I want to integrate some threat feeds and also trying out MaaS . I saw like those are done during enrichment.If we can pass a batch of data to the threat feed endpoints or MaaS endpoint, that would be great.We have a custom threat feed, i need to pass a telemetry source data to that custom feed for enrichment.It can accept 100k records for processing. So i thought if i can pass set of records to that ,it would help. Also for MaaS, I can deploy some machine learning models. So batch enrichment should be helpful.
Regards Jai On Fri, 6 Dec, 2019, 4:47 AM Nick Allen, <[email protected]> wrote: > Hi Jai - > > > Metron is a SIEM tool and by default if i add is_alert as true,then how > can i really leverage Metron as a SIEM . please correct me if i am > wrong.All data will be pushed as alerts right. > > You should set `is_alert` to a Stellar expression using whatever logic is > appropriate for your use case. This lets you determine what is important > enough for you to triage. You wouldn't just set it to true for all > telemetry, unless you do indeed want to triage everything. All telemetry > gets indexed for search (Solr/Elasticsearch) and analytics (HDFS). > > > How about batch enrichment. Is it possible to use batch during > enrichment. > > No, we do enrichment on the streaming telemetry. Keep in mind many > enrichments can be time sensitive. What sort of use case do you have in > mind for this? > > Hope this helps > > > > On Thu, Dec 5, 2019 at 11:48 AM Geeks Girls <[email protected]> > wrote: > >> Hi Sanket, >> >> >> Thanks a lot for the explanation. I am able to see the logs after adding >> an additional column is_alert to true. Metron is a SIEM tool and by default >> if i add is_alert as true,then how can i really leverage Metron as a SIEM . >> please correct me if i am wrong.All data will be pushed as alerts right. >> >> Also i could see configuration for batch indexing in json file. How about >> batch enrichment. Is it possible to use batch during enrichment. >> >> Thanks, >> Jai >> >> >> On Thu, 5 Dec, 2019, 4:14 AM Sanket Sharma, <[email protected]> >> wrote: >> >>> Hi Jai, >>> >>> >>> >>> Please see my responses below: >>> >>> >>> >>> >>>>>>>“But for bro logs, is_alert field is blank .I verified the data >>> in the Kibana. Though the is_alert is blank ,those logs also appearing in >>> metron alerts ui.How this could be possible.” >>> >>> >>> >>> This confused me in the beginning as well, but “is_alert” field only >>> controls threat triage score and does not change the behaviour of what gets >>> display in the alerts UI. If the is_alert field is set to true, then threat >>> triage score is calculated based on the rules specified. If is_alert is set >>> to false, then score calculation is skipped. It has no effect on indexing >>> and does not change the behaviour of what gets displayed in the alerts UI. >>> As @Nick Allen <[email protected]> explained to me earlier: >>> >>> >>> >>> “Threat Triage only runs on messages where there is a field named >>> "is_alert" with a Stellar expression that evaluates to true. This allows >>> you to avoid the expense of Threat Triage, in cases where you know it is >>> not needed. The "is_alert" field is used as a flag to indicate which >>> messages should undergo Threat Triage processing. Once a message completes >>> Threat Triage and gets a score, changing or removing the "is_alert" field >>> does not do anything. “ >>> >>> >>> >>> >>> >>> >>>>>>>” Also i pushed json data to the new datasource which i >>> configured . I created a new elastic search template. When reading the >>> readme file, it has been mentioned to create metron_alert field. What is >>> the difference between is_alert and metron_alert.” >>> >>> >>> >>> As explained earlier, the is_alert field is required on the incoming >>> message/event from the telemetry source. If the field is present and set to >>> true, then the event/message will be undergo threat triage processing and a >>> score will be assigned to it using the rules defined. “metron_alert” field >>> on the other hand is required on the index template on Elasticsearch (not >>> on the incoming event/message). I stand to be corrected, but there are some >>> references which mention that this is a dummy field ( >>> https://metron.apache.org/current-book/metron-platform/metron-elasticsearch/index.html >>> ) >>> >>> >>> >>> >>>>>>>” But i couldn't find anything i. AlertsUI” >>> >>> >>> >>> Usually it is because of a missing field that is required by >>> Elasticsearch and metron. You may refer to documentation here >>> https://github.com/apache/metron/tree/master/metron-platform/metron-elasticsearch/metron-elasticsearch-common >>> and >>> https://docs.cloudera.com/HDPDocuments/HCP1/HCP-1.9.0/add-new-telemetry-data-source/content/create_elasticsearch_index_template.html >>> >>> Refer to section “Elastic Search” and “Using Metron with ElasticSearch >>> 5.6” for details. Please also refer to metron-rest logs. If you still face >>> issues, please include your index template in your response. >>> >>> >>> >>> Hope that helps. >>> >>> >>> >>> Best regards, >>> >>> Sanket >>> >>> >>> >>> *From: *Geeks Girls <[email protected]> >>> *Reply to: *"[email protected]" <[email protected]> >>> *Date: *Wednesday, 04 December 2019 at 21:34 >>> *To: *"[email protected]" <[email protected]> >>> *Subject: *Data not populating in metron alerts ui >>> >>> >>> >>> Hi, >>> >>> >>> >>> I am planning to use Metron as a SIEM and exploring it's features. >>> Thanks for the great documentation. It helped a lot to set it up quickly. >>> Initially configured snort ,bro,yaf logs to flow into Metron . For snort, >>> could see threat triage rules configured in the Metron enrichment config. >>> But for bro logs, is_alert field is blank .I verified the data in the >>> Kibana. Though the is_alert is blank ,those logs also appearing in metron >>> alerts ui.How this could be possible. >>> >>> Also i pushed json data to the new datasource which i configured . I >>> created a new elastic search template. When reading the readme file, it has >>> been mentioned to create metron_alert field. What is the difference between >>> is_alert and metron_alert. >>> >>> >>> >>> What are the configuration needed to push data as alerts in metron >>> AlertsUI. I could see logs are being parsed,enriched and indexed in the >>> elastic search. So created Kibana dashboard .But i couldn't find anything >>> i. AlertsUI. What should i do?Any help is highly appreciated. >>> >>> >>> >>> Thanks, >>> >>> Jai >>> >>
