Also please consider the security of the scripts and script injection attacks. For example, we should probably restrict file access.
Thanks Carolyn On 1/3/17, 3:25 PM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: >A script bolt would still allow them to write the script the way they want >to, but would avoid having to write all the scaffolding. >The matter then would be how to integrate that script bolt into the >topologies. > > >On January 3, 2017 at 15:17:59, zeo...@gmail.com (zeo...@gmail.com) wrote: > >Right, that definitely is more efficient, but part of the point here is to >lower the barrier of entry to using Metron. > >It makes Metron's triage abilities more flexible and allows a user to reuse >existing code quickly and easily. Having this available for PoC, >prototyping, and low volume environments or situations (only when threat >score is 100, for instance) is important, as it lowers the barrier to entry >of migrating a company to a Metron environment. > >I see this as a tradeoff where I would prioritize ease of use over >efficiency. There's nothing wrong with making both options available, at >some point, and making their different use cases clear. > >Jon > >On Tue, Jan 3, 2017 at 1:47 PM Matt Foley <ma...@apache.org> wrote: > >Well, yes :-) >And clearly it should always be more efficient to write a custom bolt in >Java than to invoke a script and manage it. > >--Matt > >From: Otto Fowler <ottobackwa...@gmail.com> >Date: Tuesday, January 3, 2017 at 7:08 AM >To: "dev@metron.incubator.apache.org" <dev@metron.incubator.apache.org>, >Matt Foley <ma...@apache.org> >Subject: Re: Custom Storm Topologies > >Wouldn’t that be a bolt? > > >On January 2, 2017 at 14:39:34, Matt Foley (ma...@apache.org) wrote: >Should we consider a script calling capability that can launch a streaming >script and keep it alive and fed, long-term, rather than launching the >script anew every time the Stellar function is invoked? I’m thinking two >basic rules: Write a line, read a line; and always have a timeout. Prob >need a UID of some sort for a cache of running process objects. > >--Matt > >On 1/2/17, 8:50 AM, "Carolyn Duby" <cd...@hortonworks.com> wrote: > > >Inserting a script inline is ok for low throughput and prototyping but once >you get higher throughput (millions of events per second), it’s probably >going to be a bottleneck. > > >For Metron-571 you might want to consider a java based extension plugin >similar to Eclipse plugins. > >Thanks >Carolyn > >On 12/31/16, 5:22 PM, "Tyler Moore" <tmo...@goflyball.com> wrote: > >>Thanks Jon, >> >>I'll look over the tutorial and put something together for the SHELL_EXEC >>stellar function. >>I don't believe I have permissions to assign in Jira if you want to assign >>to me my username is devopsec. >>I'll post back details and we can review security issues >> >>Regards, >> >>Tyler Moore >>Software Engineer >>Phone: 248-909-2769 <(248)%20909-2769> >>Email: moore.ty...@goflyball.com >> >> >>On Sat, Dec 31, 2016 at 9:46 AM, zeo...@gmail.com <zeo...@gmail.com> wrote: >> >>> Casey did a tutorial on how to add your own Stellar function here >>> <https://www.youtube.com/watch?v=VAEU4JjbS1o> - there is not an existing >>> function that does this (current functions are listed here >>> <https://github.com/apache/incubator-metron/tree/master/ >>> metron-platform/metron-common#stellar-core-functions>). >>> I noticed that some of the Stellar function documentation was a bit dated >>> so I've opened a PR to update it here >>> <https://github.com/apache/incubator-metron/pull/407>. >>> >>> As this is something I need as well, I'd be happy to assist you where I >>> can. Perhaps you want to self-assign METRON-571 >>> <https://issues.apache.org/jira/browse/METRON-571>? I do have some >>> security concerns with a SHELL_EXEC function because it could result in >RCE >>> - if that's the route you go I could probably help with a thorough secure >>> code review. >>> >>> Jon >>> >>> On Fri, Dec 30, 2016 at 10:43 PM Tyler Moore <tmo...@goflyball.com> >wrote: >>> >>> Thank you everyone for your suggestions, >>> >>> I believe that kicking off the function via stellar would be the optimal >>> solution. If anyone has an example of calling external code via stellar >>> that would be very helpful. Thanks! >>> >>> Regards, >>> >>> Tyler Moore >>> IT Specialist >>> tyler.math...@yahoo.com >>> 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> >>> >>> > On Dec 30, 2016, at 17:54, Otto Fowler <ottobackwa...@gmail.com> wrote: >>> > >>> > They are all extension points. >>> > >>> >> On December 30, 2016 at 16:34:58, zeo...@gmail.com (zeo...@gmail.com) >>> wrote: >>> >> >>> >> Right but unless I'm missing something, both of those options are more >>> >> rigid and the MaaS service would have an unnecessary delay as opposed >to >>> >> doing it entirely in Stellar. Unless there's a reason to do otherwise >>> that >>> >> I'm missing, I would think doing this in Stellar gives you a more >timely >>> >> and (re)configurable end result. >>> >> >>> >> Jon >>> >> >>> >>> On Fri, Dec 30, 2016, 16:22 Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>> >>> >>> I think there are a couple of things you can do here. There way to >get >>> >>> something else into the split is to have another adapter to split to, >>> which >>> >>> is what I think you mean. You can also integrate with MaaS and create >>> a >>> >>> service that you can call via STELLAR. >>> >>> >>> >>> >>> >>> >>> >>> On December 30, 2016 at 15:08:48, Otto Fowler ( >ottobackwa...@gmail.com >>> ) >>> >>> wrote: >>> >>> >>> >>> Or a Maas service? >>> >>> >>> >>> >>> >>> On December 30, 2016 at 13:52:06, zeo...@gmail.com (zeo...@gmail.com) >>> >>> wrote: >>> >>> >>> >>> Depending on the details it sounds like a much simpler solution would >>> be >>> >>> to >>> >>> handle this in a Stellar function. >>> >>> >>> >>> Jon >>> >>> >>> >>>> On Fri, Dec 30, 2016, 13:27 Tyler Moore <tmo...@goflyball.com> >wrote: >>> >>>> >>> >>>> Happy Holidays Metron Devs! >>> >>>> >>> >>>> Could anyone lend me some guidance on customizing the storm >topologies >>> >>> in >>> >>>> metron? What I am am trying to accomplish: >>> >>>> >>> >>>> 1) Add a method to the threat intel joiner bolt that sends an http >>> post >>> >>>> with the score of the threat to a remote rest api. This will >>> >>> conditionally >>> >>>> trigger notifications based on user settings in another database >(the >>> >>>> backend processing logic is on another platform). >>> >>>> The score should be available within the JSONObject but I am not an >>> >>> expert >>> >>>> with storm and I am not completely understanding what conditions >>> >>> constitute >>> >>>> when the threat feed is considered an "alert" in metron. Please >>> clarify. >>> >>>> >>> >>>> 2) How would I add an external dependency, my http rest java class, >to >>> >>> the >>> >>>> metron maven build process? More specifically, if I was adding a >>> custom >>> >>>> class that needed accessed by a bolt in storm, how would I add this >in >>> >>>> maven as a dependency. I have limited experience with maven but, my >>> >>>> understanding is that I would add it to the pom.xml and recompile. >>> >>>> Although, the metron quick dev platform is built on a vm, would I >need >>> >>> to >>> >>>> account for this? Please advise. >>> >>>> >>> >>>> Regards, >>> >>>> >>> >>>> Tyler Moore >>> >>>> >>> >>>> >>> >>>> Software Engineer >>> >>>> Phone: 248-909-2769 <(248)%20909-2769> <(248)%20909-2769> >>> >>>> Email: moore.ty...@goflyball.com >>> >>>> >>> >>> -- >>> >>> >>> >>> Jon >>> >>> >>> >>> Sent from my mobile device >>> >>> >>> >>> -- >>> >> >>> >> Jon >>> >> >>> >> Sent from my mobile device >>> >> >>> >>> -- >>> >>> Jon >>> >>> Sent from my mobile device >>> > > > > > >-- > >Jon > >Sent from my mobile device
