Would it make sense to lean on something like Apache NiFi for this? It seems a good fit to handle getting data from wherever (web service, poll, push etc, streams etc). If we were to build a processor which encapsulated the threat intel loader logic, that would provide a granular route to update threat intel entries in a more streaming manner. We could of course do the same thing in code with storm topologies, but I would wonder whether threat intel feeds would have enough volume to require that.
Simon > On 16 Feb 2018, at 07:11, Ali Nazemian <alinazem...@gmail.com> wrote: > > I think one of the challenges is where the scope of threat intel ends from > the Metron roadmap? Does it gonna relly on supporting a standard format and > a loader to send it to HBase for the later threat intel use cases? > > In my opinion, it would be better to have a separate topology (sort of > similar to the profiler approach) to get the feeds (maybe from Kafka) and > load it into HBase frequently based on what criteria we want to have. Maybe > we need to have some normalizations for the threat feeds (either aggregated > or single feed) as an example (or any other transformation by using > Stellar). Maybe we need to tailor row_key in a way that can be utilised > based on the threat intel look up we want to have further from the > enrichment topology. The problem I see with different loaders in Metron is > we can mostly use them for the purpose of POC, but if you want to build an > actual use case for a production platform then it will be out of the > flexibility of a loader, so we will end up feeding data to HBase based on > our use case. > > In this case, maybe it won't be very important we want to use an aggregator > X or aggregator Y, we can integrate it with Metron based on integration > points. > > Cheers, > Ali > > On Wed, Feb 14, 2018 at 11:28 PM, Simon Elliston Ball < > si...@simonellistonball.com> wrote: > >> We used to install soltra edge in the old ansible builds (which have >> thankfully now been pared back in the interests of stability in full dev). >> Soltra has not been a good option since they went proprietary, so since >> then we’ve included opentaxii (BSD 3) as a discovery and aggregator. >> >> Most of the challenges are around licensing. Hippocampe is part of The >> Hive Project, which is AGPL, which is an apache category X license so can’t >> be included. >> >> Mindmeld is much better license-wise (Apache 2) so would be well worth >> community consideration. I kinda like it as a framework, but >> >> I for one would be very pleased to hear a broader community discussion >> around which platforms we should have integrations with via the threat >> intel loader, or even through a direct to hbase streaming connector. >> >> Simon >> >>> On 14 Feb 2018, at 03:13, Ali Nazemian <alinazem...@gmail.com> wrote: >>> >>> Hi All, >>> >>> I would like to understand Metron community view on Threat Intel >>> aggregators as well as the roadmap of threat intelligence and threat >>> hunting. There are some open source options available regarding threat >>> intel aggregator such as Minemeld, Hippocampe, etc. Is there any plan to >>> build that as a part of Metron in future? Is there any specific >> aggregator >>> you think would be more aligned with Metron roadmap? >>> >>> Cheers, >>> Ali >> >> > > > -- > A.Nazemian
