We use a guava cache to cache the data for 24 hours. You can see how it's done here: https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/stellar/ObjectGet.java
We also do something like this in GEO_GET as well, but it's a bit more complex. On Fri, Feb 2, 2018 at 9:35 AM, Simon Elliston Ball < si...@simonellistonball.com> wrote: > I forgot we added OBJECT_GET. How does the caching work on that? > > Simn > > > On 2 Feb 2018, at 14:33, Nick Allen <n...@nickallen.org> wrote: > > There are many functions that use the global configuration. For example, > GET_GEO in org.apache.metron.enrichment.stellar.GeoEnrichmentFunctions. > There might be a better example, but that is one is staring at me at the > moment. > > There is an OBJECT_GET function defined in > org.apache.metron.enrichment.stellar.ObjectGet > that was purpose-built to retrieve files from HDFS. If you wanted to > retrieve a configuration from HDFS that would be a good example (if you > can't just use that functions directly). > > On Fri, Feb 2, 2018 at 8:50 AM Ali Nazemian <alinazem...@gmail.com> wrote: > >> Is there any Stellar function already been implemented in Metron that has >> a config file associated with it? I am trying to get an idea of how it >> works. >> >> On 3 Feb. 2018 00:44, "Simon Elliston Ball" <si...@simonellistonball.com> >> wrote: >> >>> Depends how you write the function class, but most likely, yes. Hence >>> global config option. >>> >>> Simon >>> >>> On 2 Feb 2018, at 13:42, Ali Nazemian <alinazem...@gmail.com> wrote: >>> >>> Does it mean every time the function gets called it will load the >>> config, but if I use the global one it will only read it one time and it >>> will be available in memory? >>> >>> On 2 Feb. 2018 21:53, "Simon Elliston Ball" <si...@simonellistonball.com> >>> wrote: >>> >>>> Shouldn’t be. The one this I would point out though is that you don’t >>>> necessarily know which supervisor you will be running from, so pulling from >>>> HDFS would make sense. That said, the performance implications are probably >>>> not great. A good option here would be to have the config available in the >>>> global config for example and refer to that, since most instances of >>>> stellar apply global config to their context. >>>> >>>> Simon >>>> >>>> >>>> On 2 Feb 2018, at 07:14, Ali Nazemian <alinazem...@gmail.com> wrote: >>>> >>>> Will be any problem if the Stellar function we want to implement need >>>> to load an external config file? >>>> >>>> Cheers, >>>> Ali >>>> >>>> On Thu, Jan 18, 2018 at 4:58 PM, Ali Nazemian <alinazem...@gmail.com> >>>> wrote: >>>> >>>>> Thanks, All. >>>>> >>>>> Yes, Nick. It is highly related to our use case and the way that we >>>>> are going to enrich events with assets and vulnerability properties. It is >>>>> not a general case at all. >>>>> >>>>> Cheers, >>>>> Ali >>>>> >>>>> On Thu, Jan 18, 2018 at 5:43 AM, Matt Foley <ma...@apache.org> wrote: >>>>> >>>>>> Besides the example code Simon mentioned at >>>>>> https://github.com/apache/metron/tree/master/metron- >>>>>> stellar/stellar-3rd-party-example , >>>>>> there is some documentation at http://metron.apache.org/ >>>>>> current-book/metron-stellar/stellar-common/3rdPartyStellar.html >>>>>> >>>>>> >>>>>> >>>>>> *From: *Nick Allen <n...@nickallen.org> >>>>>> *Reply-To: *"user@metron.apache.org" <user@metron.apache.org> >>>>>> *Date: *Wednesday, January 17, 2018 at 4:46 AM >>>>>> *To: *"user@metron.apache.org" <user@metron.apache.org> >>>>>> *Subject: *Re: Define a function that can be used in Stellar >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> If something we have already does not fit the bill, I would recommend >>>>>> creating that function in Java. Since you described it as "a bit >>>>>> complex" >>>>>> and "the logic would be complicated" I don't see any value in defining >>>>>> something like this in Stellar with named functions. >>>>>> >>>>>> >>>>>> >>>>>> Best >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 17, 2018 at 7:38 AM Simon Elliston Ball < >>>>>> si...@simonellistonball.com> wrote: >>>>>> >>>>>> Have you looked at the recent TLSH functions in Stellar? We already >>>>>> have that for similarity preserving hashes. >>>>>> >>>>>> >>>>>> >>>>>> Simon >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 17 Jan 2018, at 12:35, Ali Nazemian <alinazem...@gmail.com> wrote: >>>>>> >>>>>> It is a bit complex. We want to create a function that accepts a list >>>>>> of arguments for an asset and generate an asset identifier that can be >>>>>> used >>>>>> as a row_key for the enrichment store. The logic would be complicated, >>>>>> though. We may need to include some sort of similarity aware hash >>>>>> function >>>>>> as a part of this custom function. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 17, 2018 at 10:32 PM, Nick Allen <n...@nickallen.org> >>>>>> wrote: >>>>>> >>>>>> Ali - Can you describe the logic that you are trying to perform? That >>>>>> would be useful as a use case to help drive a discussion around creating >>>>>> named functions in Stellar. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 17, 2018 at 6:29 AM Ali Nazemian <alinazem...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Thanks, Simon. We have already got a script to deal with classpath >>>>>> management for the parsers. We should be able to use it for this >>>>>> extension >>>>>> as well. >>>>>> >>>>>> >>>>>> >>>>>> Yeah, I agree. It will be much easier to define functions on the fly >>>>>> and use them afterwards. It could be defined as Lambda or custom >>>>>> function. >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> Ali >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 17, 2018 at 9:42 PM, Simon Elliston Ball < >>>>>> si...@simonellistonball.com> wrote: >>>>>> >>>>>> https://github.com/apache/metron/tree/master/metron- >>>>>> stellar/stellar-3rd-party-example gives good details on how to add a >>>>>> stellar function. >>>>>> >>>>>> >>>>>> >>>>>> Stellar will pick up an annotated function on its class path, so to >>>>>> add function there is no need to rebuild metron module, but you do need >>>>>> your modules on the classpath, and, pending 777, to deal with things like >>>>>> class path clash in your dependencies. >>>>>> >>>>>> >>>>>> >>>>>> Another idea worth discussion on the dev list is probably the notion >>>>>> of defining stellar functions in stellar, which would be a much simpler >>>>>> solution than custom java functions if you can already express you logic >>>>>> in >>>>>> stellar. >>>>>> >>>>>> >>>>>> >>>>>> Simon >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 17 Jan 2018, at 10:37, Ali Nazemian <alinazem...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> Hi Simon, >>>>>> >>>>>> >>>>>> >>>>>> Yes, that is exactly what we are looking for. Is there any example >>>>>> regarding adding a Stellar function in Java? Hopefully, we don't need to >>>>>> rebuild the corresponding modules for this? >>>>>> >>>>>> >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Ali >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Jan 17, 2018 at 8:40 PM, Simon Elliston Ball < >>>>>> si...@simonellistonball.com> wrote: >>>>>> >>>>>> At present you can certainly create custom stellar functions in Java. >>>>>> I’m guessing however that what you’re looking to do is create a kind of >>>>>> function that combines a number of stellar functions to avoid repetition, >>>>>> or to ensure consistency of certain parameters for example. Is that what >>>>>> you’re looking for? Maybe some sort of syntax to create a named stellar >>>>>> function similar to the way we create lambdas? >>>>>> >>>>>> Simon >>>>>> >>>>>> >>>>>> > On 17 Jan 2018, at 07:25, Ali Nazemian <alinazem...@gmail.com> >>>>>> wrote: >>>>>> > >>>>>> > Hi all, >>>>>> > >>>>>> > Is there any way that we can define a function that can be used >>>>>> rather than duplicating a logic multiple times? >>>>>> > >>>>>> > Cheers, >>>>>> > Ali >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> A.Nazemian >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> A.Nazemian >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> A.Nazemian >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> A.Nazemian >>>>> >>>> >>>> >>>> >>>> -- >>>> A.Nazemian >>>> >>>> >>>> >>> >
