Right, that definitely is more efficient, but part of the point here is to lower the barrier of entry to using Metron.
It makes Metron's triage abilities more flexible and allows a user to reuse existing code quickly and easily. Having this available for PoC, prototyping, and low volume environments or situations (only when threat score is 100, for instance) is important, as it lowers the barrier to entry of migrating a company to a Metron environment. I see this as a tradeoff where I would prioritize ease of use over efficiency. There's nothing wrong with making both options available, at some point, and making their different use cases clear. Jon On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote: Well, yes :-) And clearly it should always be more efficient to write a custom bolt in Java than to invoke a script and manage it. --Matt From: Otto Fowler <ottobackwa...@gmail.com> Date: Tuesday, January 3, 2017 at 7:08 AM To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org>, Matt Foley <ma...@apache.org> Subject: Re: Custom Storm Topologies Wouldn’t that be a bolt? On January 2, 2017 at 14:39:34, Matt Foley (ma...@apache.org) wrote: Should we consider a script calling capability that can launch a streaming script and keep it alive and fed, long-term, rather than launching the script anew every time the Stellar function is invoked? I’m thinking two basic rules: Write a line, read a line; and always have a timeout. Prob need a UID of some sort for a cache of running process objects. --Matt On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote: Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck. For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins. Thanks Carolyn On 12/31/16, 5:22 PM, "Tyler Moore" <tmo...@goflyball.com> wrote: >Thanks Jon, > >I'll look over the tutorial and put something together for the SHELL_EXEC >stellar function. >I don't believe I have permissions to assign in Jira if you want to assign >to me my username is devopsec. >I'll post back details and we can review security issues > >Regards, > >Tyler Moore >Software Engineer >Phone: 248-909-2769 <(248)%20909-2769> >Email: moore.ty...@goflyball.com > > >On Sat, Dec 31, 2016 at 9:46 AM, zeo...@gmail.com <zeo...@gmail.com> wrote: > >> Casey did a tutorial on how to add your own Stellar function here >> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing >> function that does this (current functions are listed here >> <https://github.com/apache/incubator-metron/tree/master/ >> metron-platform/metron-common#stellar-core-functions>). >> I noticed that some of the Stellar function documentation was a bit dated >> so I've opened a PR to update it here >> <https://github.com/apache/incubator-metron/pull/407>. >> >> As this is something I need as well, I'd be happy to assist you where I >> can. Perhaps you want to self-assign METRON-571 >> <https://issues.apache.org/jira/browse/METRON-571>? I do have some >> security concerns with a SHELL_EXEC function because it could result in RCE >> - if that's the route you go I could probably help with a thorough secure >> code review. >> >> Jon >> >> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmo...@goflyball.com> wrote: >> >> Thank you everyone for your suggestions, >> >> I believe that kicking off the function via stellar would be the optimal >> solution. If anyone has an example of calling external code via stellar >> that would be very helpful. Thanks! >> >> Regards, >> >> Tyler Moore >> IT Specialist >> tyler.math...@yahoo.com >> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> >> >> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwa...@gmail.com> wrote: >> > >> > They are all extension points. >> > >> >> On December 30, 2016 at 16:34:58, zeo...@gmail.com (zeo...@gmail.com) >> wrote: >> >> >> >> Right but unless I'm missing something, both of those options are more >> >> rigid and the MaaS service would have an unnecessary delay as opposed to >> >> doing it entirely in Stellar. Unless there's a reason to do otherwise >> that >> >> I'm missing, I would think doing this in Stellar gives you a more timely >> >> and (re)configurable end result. >> >> >> >> Jon >> >> >> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> >> >>> I think there are a couple of things you can do here. There way to get >> >>> something else into the split is to have another adapter to split to, >> which >> >>> is what I think you mean. You can also integrate with MaaS and create >> a >> >>> service that you can call via STELLAR. >> >>> >> >>> >> >>> >> >>> On December 30, 2016 at 15:08:48, Otto Fowler ( ottobackwa...@gmail.com >> ) >> >>> wrote: >> >>> >> >>> Or a Maas service? >> >>> >> >>> >> >>> On December 30, 2016 at 13:52:06, zeo...@gmail.com (zeo...@gmail.com) >> >>> wrote: >> >>> >> >>> Depending on the details it sounds like a much simpler solution would >> be >> >>> to >> >>> handle this in a Stellar function. >> >>> >> >>> Jon >> >>> >> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmo...@goflyball.com> wrote: >> >>>> >> >>>> Happy Holidays Metron Devs! >> >>>> >> >>>> Could anyone lend me some guidance on customizing the storm topologies >> >>> in >> >>>> metron? What I am am trying to accomplish: >> >>>> >> >>>> 1) Add a method to the threat intel joiner bolt that sends an http >> post >> >>>> with the score of the threat to a remote rest api. This will >> >>> conditionally >> >>>> trigger notifications based on user settings in another database (the >> >>>> backend processing logic is on another platform). >> >>>> The score should be available within the JSONObject but I am not an >> >>> expert >> >>>> with storm and I am not completely understanding what conditions >> >>> constitute >> >>>> when the threat feed is considered an "alert" in metron. Please >> clarify. >> >>>> >> >>>> 2) How would I add an external dependency, my http rest java class, to >> >>> the >> >>>> metron maven build process? More specifically, if I was adding a >> custom >> >>>> class that needed accessed by a bolt in storm, how would I add this in >> >>>> maven as a dependency. I have limited experience with maven but, my >> >>>> understanding is that I would add it to the pom.xml and recompile. >> >>>> Although, the metron quick dev platform is built on a vm, would I need >> >>> to >> >>>> account for this? Please advise. >> >>>> >> >>>> Regards, >> >>>> >> >>>> Tyler Moore >> >>>> >> >>>> >> >>>> Software Engineer >> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> >> >>>> Email: moore.ty...@goflyball.com >> >>>> >> >>> -- >> >>> >> >>> Jon >> >>> >> >>> Sent from my mobile device >> >>> >> >>> -- >> >> >> >> Jon >> >> >> >> Sent from my mobile device >> >> >> >> -- >> >> Jon >> >> Sent from my mobile device >> -- Jon Sent from my mobile device
