Well, yes :-) And clearly it should always be more efficient to write a custom bolt in Java than to invoke a script and manage it.
--Matt From: Otto Fowler <ottobackwa...@gmail.com> Date: Tuesday, January 3, 2017 at 7:08 AM To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org>, Matt Foley <ma...@apache.org> Subject: Re: Custom Storm Topologies Wouldn’t that be a bolt? On January 2, 2017 at 14:39:34, Matt Foley (ma...@apache.org) wrote: Should we consider a script calling capability that can launch a streaming script and keep it alive and fed, long-term, rather than launching the script anew every time the Stellar function is invoked? I’m thinking two basic rules: Write a line, read a line; and always have a timeout. Prob need a UID of some sort for a cache of running process objects. --Matt On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote: Inserting a script inline is ok for low throughput and prototyping but once you get higher throughput (millions of events per second), it’s probably going to be a bottleneck. For Metron-571 you might want to consider a java based extension plugin similar to Eclipse plugins. Thanks Carolyn On 12/31/16, 5:22 PM, "Tyler Moore" <tmo...@goflyball.com> wrote: >Thanks Jon, > >I'll look over the tutorial and put something together for the SHELL_EXEC >stellar function. >I don't believe I have permissions to assign in Jira if you want to assign >to me my username is devopsec. >I'll post back details and we can review security issues > >Regards, > >Tyler Moore >Software Engineer >Phone: 248-909-2769 >Email: moore.ty...@goflyball.com > > >On Sat, Dec 31, 2016 at 9:46 AM, zeo...@gmail.com <zeo...@gmail.com> wrote: > >> Casey did a tutorial on how to add your own Stellar function here >> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing >> function that does this (current functions are listed here >> <https://github.com/apache/incubator-metron/tree/master/ >> metron-platform/metron-common#stellar-core-functions>). >> I noticed that some of the Stellar function documentation was a bit dated >> so I've opened a PR to update it here >> <https://github.com/apache/incubator-metron/pull/407>. >> >> As this is something I need as well, I'd be happy to assist you where I >> can. Perhaps you want to self-assign METRON-571 >> <https://issues.apache.org/jira/browse/METRON-571>? I do have some >> security concerns with a SHELL_EXEC function because it could result in RCE >> - if that's the route you go I could probably help with a thorough secure >> code review. >> >> Jon >> >> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmo...@goflyball.com> wrote: >> >> Thank you everyone for your suggestions, >> >> I believe that kicking off the function via stellar would be the optimal >> solution. If anyone has an example of calling external code via stellar >> that would be very helpful. Thanks! >> >> Regards, >> >> Tyler Moore >> IT Specialist >> tyler.math...@yahoo.com >> 248-909-2769 <(248)%20909-2769> >> >> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwa...@gmail.com> wrote: >> > >> > They are all extension points. >> > >> >> On December 30, 2016 at 16:34:58, zeo...@gmail.com (zeo...@gmail.com) >> wrote: >> >> >> >> Right but unless I'm missing something, both of those options are more >> >> rigid and the MaaS service would have an unnecessary delay as opposed to >> >> doing it entirely in Stellar. Unless there's a reason to do otherwise >> that >> >> I'm missing, I would think doing this in Stellar gives you a more timely >> >> and (re)configurable end result. >> >> >> >> Jon >> >> >> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> >> >>> I think there are a couple of things you can do here. There way to get >> >>> something else into the split is to have another adapter to split to, >> which >> >>> is what I think you mean. You can also integrate with MaaS and create >> a >> >>> service that you can call via STELLAR. >> >>> >> >>> >> >>> >> >>> On December 30, 2016 at 15:08:48, Otto Fowler (ottobackwa...@gmail.com >> ) >> >>> wrote: >> >>> >> >>> Or a Maas service? >> >>> >> >>> >> >>> On December 30, 2016 at 13:52:06, zeo...@gmail.com (zeo...@gmail.com) >> >>> wrote: >> >>> >> >>> Depending on the details it sounds like a much simpler solution would >> be >> >>> to >> >>> handle this in a Stellar function. >> >>> >> >>> Jon >> >>> >> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmo...@goflyball.com> wrote: >> >>>> >> >>>> Happy Holidays Metron Devs! >> >>>> >> >>>> Could anyone lend me some guidance on customizing the storm topologies >> >>> in >> >>>> metron? What I am am trying to accomplish: >> >>>> >> >>>> 1) Add a method to the threat intel joiner bolt that sends an http >> post >> >>>> with the score of the threat to a remote rest api. This will >> >>> conditionally >> >>>> trigger notifications based on user settings in another database (the >> >>>> backend processing logic is on another platform). >> >>>> The score should be available within the JSONObject but I am not an >> >>> expert >> >>>> with storm and I am not completely understanding what conditions >> >>> constitute >> >>>> when the threat feed is considered an "alert" in metron. Please >> clarify. >> >>>> >> >>>> 2) How would I add an external dependency, my http rest java class, to >> >>> the >> >>>> metron maven build process? More specifically, if I was adding a >> custom >> >>>> class that needed accessed by a bolt in storm, how would I add this in >> >>>> maven as a dependency. I have limited experience with maven but, my >> >>>> understanding is that I would add it to the pom.xml and recompile. >> >>>> Although, the metron quick dev platform is built on a vm, would I need >> >>> to >> >>>> account for this? Please advise. >> >>>> >> >>>> Regards, >> >>>> >> >>>> Tyler Moore >> >>>> >> >>>> >> >>>> Software Engineer >> >>>> Phone: 248-909-2769 <(248)%20909-2769> >> >>>> Email: moore.ty...@goflyball.com >> >>>> >> >>> -- >> >>> >> >>> Jon >> >>> >> >>> Sent from my mobile device >> >>> >> >>> -- >> >> >> >> Jon >> >> >> >> Sent from my mobile device >> >> >> >> -- >> >> Jon >> >> Sent from my mobile device >>
