I think my point is that maybe we should have a discuss about: * PCAP UI, goals etc * Where it would live and why, what that would mean etc * Backend ( this original mail )
On May 3, 2018 at 18:34:00, Michael Miklavcic (michael.miklav...@gmail.com) wrote: Otto, what are you and your customers finding useful and/or difficult from a split management/alerts UI perspective? It might help us to restate the original scope and intent around maintaining separate management and alert UI's, to your point about "contrary to previous direction." I personally don't have a strong position on this other than 1) management is a different feature set from drilling into threat intel, yet many apps still have their management UI combined with the end user experience and 2) we should probably consider pcap in context of a workflow with alerts. On Thu, May 3, 2018 at 4:19 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > If that UI becomes the Alerts _and_ the PCAP Query UI, then it isn’t the > alerts ui anymore. > > It is becoming more of a “composite” app, with multiple feature ui’s > together. I didn’t think that > was what we were going for, thus the config ui and the alert ui. > > Just adding disparate thing as ‘new tabs’ to a ui may be expedient but it > seems contrary to > our previous direction. > > There are a few things to consider if we are going to start moving > everything into Alerts Ui aren’t there? > > It may be a better road to bring it in on it’s own like the alerts ui > effort, so it can be released with ‘qualifiers’ and tested with > the right expectations without effecting the Alerts UI. > > > > On May 3, 2018 at 17:25:54, Ryan Merriman (merrim...@gmail.com) wrote: > > Otto, > > I'm assuming just adding it to the Alerts UI is less work but I wouldn't be > strongly opposed to it being it's own UI. What are the reasons for doing > that? > > Mike, > > On using metron-api: > > 1. I'm making an assumption about it not being used much. Maybe it > still works without issue. I agree, we'll have to test anything we build > so this is a minor issue. > 2. Updating metron-api to be asynchronous is a requirement in my opinion > 3. The MPack work is the major drawback for me. We're essentially > creating a brand new Metron component. There are a lot of examples we can > draw from but it's going to be a large chunk of new MPack code to maintain > and MPack development has been painful in the past. I think it will > include: > 1. Creating a start script > 2. Creating master.py and commands.py scripts for managing the > application lifecycle, service checks, etc > 3. Creating an -env.xml file for exposing properties in Ambari > 4. Adding the component to the various MPack files > (metron_theme.json, metainfo.xml, service_advisor.py, etc.) > 4. Our Storm topologies are completely different use cases and much more > complex so I don't understand the comparison. But if you prefer this > coding style then I think this is a minor issue as well. > > On micro-services: > > 1. Our REST service already includes a lot of dependencies and is > difficult to manage in it's current state. I just went through this on > https://github.com/apache/metron/pull/1008. It was painful. When we > tried to include mapreduce and yarn dependencies it became what seemed like > an endless NoSuchMethod, NoClassDef and similar errors. Even if we can get > it to work it's going to make managing our REST service that much harder > than it already is. I think the shaded jars are the source of all this > trouble and I agree it would be nice to improve our architecture in this > area. However I don't think it's a simple fix and now we're getting into > the "will likely take a long time to plan and implement" concern. If > anyone has ideas on how to solve our shaded jar challenge I would be all > for it. > 2. All the MPack work listed above would also be required here. A > micro-services pattern is a significant shift and can't even give you > concrete examples of what exactly we would have to do. We would need to go > through extensive design and planning to even get to that point. > 3. It would be a branch new component. See above plus any new > infrastructure we would need (web server/proxy, service discovery, etc) > > On pcap-query: > > 1. I don't recall any users or customers directly using metron-api but > if you say so I believe you :) > 2. As I understand it the pcap topology and pcap query are somewhat > decoupled. Maybe location of pcap files would be shared? MPack work here > is likely to include adding a couple properties and moving some around so > they can be shared. Deciding between Ambari and global config would be > similar to properties we add to any component. > > I think you may be underestimating how difficult it's going to be to solve > our dependency problem. Or maybe it's me that is overestimating it :) It > could be something we experiment with before we start on the pcap work. > There is major upside and it would benefit the whole project. But until > then we can't fit anymore more screwdrivers in the toolbox. For me the > only reasonable options are to use the existing metron-api as it's own > separate service or call out to the pcap_query.sh script from our existing > REST app. I could go either way really. I'm just not excited about all > the MPack code we have to write for a new component. Maybe it won't be > that bad. > > On Thu, May 3, 2018 at 2:50 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > First thought is why the Alerts-UI and Not a dedicated Query UI? > > > > > > On May 3, 2018 at 14:36:04, Ryan Merriman (merrim...@gmail.com) wrote: > > > > We are planning on adding the pcap query feature to the Alerts UI. Before > > we start this work, I think it is important to get community buy in on > the > > architectural approach. There are a couple different options. > > > > One option is to leverage the existing metron-api module that exposes > pcap > > queries through a REST service. The upsides are: > > > > - some work has already been done > > - it's part of our build so we know unit and integration tests pass > > > > The downsides are: > > > > - It hasn't been used in a while and will need some end to end testing > > to make sure it still functions properly > > - It is synchronous and will block the UI, using up the limited number > > of concurrent connections available in a browser > > - It will require significant MPack work to properly set it up on install > > - It is a legacy module from OpenSOC and coding style is significantly > > different > > > > Another option would be moving to a micro-services architecture. We have > > experimented with a proof of concept and found it was too hard to add > this > > feature into our existing REST services because of all the dependencies > > that must coexist in the same application. The upsides are: > > > > - Would provide a platform for future Batch/MR/YARN type features > > - There would be fewer technical compromises since we are building it > > from the ground up > > > > The downsides are: > > > > - Will require the most effort and will likely take a long time to plan > > and implement > > - Like the previous option, will require significant MPack work > > > > A third option would be to add an endpoint to our existing REST service > > that delegates to the pcap_query.sh script through the Java Process > class. > > The upsides to this approach are: > > > > - We know the pcap_query.sh script works and would require minimal > > changes > > - Minimal MPack work is required since our REST service is already > > included > > > > The downsides are: > > > > - Does not set us up to easily add other batch-oriented features in the > > future > > - OS-level security becomes a concern since we are delegating to a > > script in a separate process > > > > I feel like ultimately we want to transition to a micro-services > > architecture because it will provide more flexibility and make it easier > > to > > grow our set of features. But in the meantime, wrapping the pcap_query.sh > > script would allow us to add this feature with less work and fewer lines > > of > > code. If and when we decide to deploy a separate REST application for > > batch features, the UI portion would require minimal changes. > > > > What does everyone think? > > > > >
