That's fantastic, thanks for that detail. Also, I'm in agreement with the recent comments from Otto and Simon.
On Fri, Nov 16, 2018 at 9:49 AM Ryan Merriman <merrim...@gmail.com> wrote: > I was still able to spin up the UI locally and debug in my testing. I am > in complete agreement, we need to ensure the developer experience doesn't > change. > > On Fri, Nov 16, 2018 at 10:47 AM Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > > > Ryan, what's remote debugging look like for UI testing with Knox enabled? > > Anything we lose from a dev testability standpoint? The discussion of > > defaults sounds reasonable to me, and I'd like to understand any other > > tradeoffs there may be for non-prod deployments like full dev. > > > > On Fri, Nov 16, 2018 at 7:20 AM Ryan Merriman <merrim...@gmail.com> > wrote: > > > > > Most of the research I've done around adding Metron as a Knox service > is > > > based on how other projects do it. The documentation is not easy to > > follow > > > so I learned by reading other service definition files. The assumption > > > that we are doing things drastically different is false. > > > > > > I completely agree with Simon. Why would we want to be dependent on > > Knox's > > > release cycle? How does that benefit us? It may reduce some > operational > > > complexity but it makes our install process more complicated because we > > > require a certain version of Knox (who knows when that gets released). > > > What do we do in the meantime? I would also like to point out that > > Metron > > > is inherently different than other Hadoop stack services. We are a > > > full-blown application with multiple UIs so the way we expose services > > > through Knox may be a little different. > > > > > > I think this will be easier to discuss when we can all see what is > > actually > > > involved. I am working on a PR that adds Metron as a Knox service and > > will > > > have that out soon. That should give everyone more context. > > > > > > On Fri, Nov 16, 2018 at 7:39 AM Simon Elliston Ball < > > > si...@simonellistonball.com> wrote: > > > > > > > You could say the same thing about Ambari, but that provides mpacks. > > Knox > > > > is also designed to be extensible through Knox service stacks since > > they > > > > realized they can’t support every project. The challenge is that the > > docs > > > > have not made it as easy as they could for the ecosystem to plug into > > > Knox, > > > > which has led to some confusion around this being a recommended > pattern > > > > (which it is). > > > > > > > > The danger of trying to get your bits into Knox is that that ties you > > to > > > > their release cycle (a problem Ambari has felt hard, hence their > > > community > > > > is moving away from the everything inside model towards everything is > > an > > > > mpack). > > > > > > > > A number of implementations of Knox also use the approach Ryan is > > > > suggesting for their own organization specific end points, so it’s > not > > > like > > > > this is an uncommon, or anti-pattern, it’s more the way Knox is > > designed > > > to > > > > work in the future, than the legacy of it only being able to handle a > > > > subset of Hadoop projects. > > > > > > > > Knox remains optional In our scenario, but we keep control over the > > > > shipping of things like rewrite rules, which allows Metron to control > > its > > > > release destiny should things like url patterns in the ui need to > > change > > > > (with a new release of angular / new module / new rest endpoint etc) > > > > instead of making a Metron release dependent on a Knox release. > > > > > > > > Imagine how we would have done with the Ambari side if we’d had to > wait > > > > for them to release every time we needed to change something in the > > > > mpack... we don’t want that happening with Knox. > > > > > > > > Simon > > > > > > > > > On 16 Nov 2018, at 13:22, Otto Fowler <ottobackwa...@gmail.com> > > wrote: > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/KNOX-841?jql=project%20%3D%20KNOX%20AND%20text%20~%20support > > > > > > > > > > Solr is angular for example. > > > > > > > > > > > > > > > On November 16, 2018 at 08:12:55, Otto Fowler ( > > ottobackwa...@gmail.com > > > ) > > > > > wrote: > > > > > > > > > > Ok, here is something I don’t understand, but would like to. > > > > > > > > > > Knox comes configured with build in services for a number of other > > > apache > > > > > products and UI’s. > > > > > It would seem to me, that the best integration with Knox would be > to > > do > > > > > what these other products have done. > > > > > > > > > > > > > > > 1. Do whatever you have to do to make your own stuff compatible. > > > > > 2. Create a knox service definition and provide it or try to get it > > > into > > > > > knox itself > > > > > > > > > > This would make the knox integration with metron optional and > > pluggable > > > > > wouldn’t it? > > > > > > > > > > Then knox with metron would just be the same as knox with anything > > > else. > > > > > Please help me if I am wrong, but we seem to be going our own way > > here. > > > > > Why don’t we just do what these other products have done? > > > > > Why don’t we try to get apache metron services accepted to the knox > > > > > project? Why don’t we model our knox integration with how XYZ does > > it? > > > > > Have we looked at how others integrate? Having all the code and > > being > > > > > able to track stuff is kind of the point of this whole thing isn’t > > it? > > > > > > > > > > Maybe this is implied and I’m missing it, if so I apologize. > > > > > > > > > > I think consistency with the rest of the hadoop stack with knox > helps > > > us. > > > > > > > > > > > > > > > > > > > > On November 15, 2018 at 22:20:00, Ryan Merriman ( > merrim...@gmail.com > > ) > > > > wrote: > > > > > > > > > > 1) Sorry I misspoke. I meant to say this is not possible in the > > Alerts > > > UI > > > > > as far as I know. I put up a PR with a proposed solution here: > > > > > https://github.com/apache/metron/pull/1266. > > > > > 2) Yes Knox is a service you can install with Ambari, similar to > > Ranger > > > > or > > > > > Spark. There are some things that are specifically configured in > Knox > > > and > > > > > there are some things specific to Metron. I will put up a PR with > the > > > > > changes needed so you can see exactly what is involved. > > > > > 3) I don't understand what you mean here. Is this a question? > > > > > 4) I think it's a little early to predict the Ambari changes > > required. > > > > > This will depend on how tasks 1-3 go. I imagine it's similar to > other > > > > > mpack work: expose some parameters in ambari and bind those to > config > > > > > files. My understanding from this thread so far is that we should > > focus > > > > on > > > > > a manual, documented approach to start. > > > > > > > > > > On Thu, Nov 15, 2018 at 7:53 PM Michael Miklavcic < > > > > > michael.miklav...@gmail.com> wrote: > > > > > > > > > >> Thanks Ryan, > > > > >> > > > > >> 1) Can you clarify "not a good way to do this?" Are you saying we > > > don't > > > > >> have a way to set this and need to add the config option, or that > a > > > > >> solution is not obvious and it's unclear what to do? It seems to > me > > > > you're > > > > >> saying the former, but I'd like to be sure. > > > > >> 2) Is Knox not a service made available by Ambari similar to > Ranger > > or > > > > >> Spark? I'm assuming that similar to Kerberos, there are some > things > > > that > > > > >> are specifically configured in Knox and others that are > > app-specific. > > > > Some > > > > >> explanation of what this looks like would be helpful. > > > > >> 3) Sounds like this follows pretty naturally from #1 > > > > >> 4) Relates to #2. I think we need some guidance on what a manual > vs > > > > >> MPack/automated install would look like. > > > > >> > > > > >> Cheers, > > > > >> Mike > > > > >> > > > > >> > > > > >>> On Thu, Nov 15, 2018 at 4:07 PM Ryan Merriman < > merrim...@gmail.com > > > > > > > wrote: > > > > >>> > > > > >>> Wanted to give an update on the context path issue. I > investigated > > > > >>> rewriting url links in the outgoing static assets with Knox and > it > > > was > > > > >> not > > > > >>> trivial. Fortunately I found a simple solution that works with or > > > > >> without > > > > >>> Knox. I changed the base tag in index.html from <base href="/"> > to > > > > <base > > > > >>> href="./">, or in other words made the base href relative. > > > > >>> > > > > >>> I believe I am at the point where I can task this out and > provide a > > > > high > > > > >>> level overview of the changes needed. I think that each task will > > be > > > a > > > > >>> manageable size and can stand alone so I don't think we need a > > > feature > > > > >>> branch. > > > > >>> > > > > >>> The first task involves a general change to the UI code. We need > a > > > way > > > > >> to > > > > >>> set the path to the REST service with a configuration setting > > because > > > > it > > > > >> is > > > > >>> different with and without Knox. Currently there is not a good > way > > to > > > > do > > > > >>> this in the UI. We can use the environment files but that is a > > build > > > > >> time > > > > >>> setting and is not flexible. I can see this capability being > useful > > > for > > > > >>> other use cases in the future. I think we could even split this > up > > > into > > > > >> 2 > > > > >>> separate tasks, one for the alerts UI and one for the management > > UI. > > > > >>> > > > > >>> The second task involves adding Knox to our stack either by > default > > > as > > > > a > > > > >>> dependency in the mpack or with a documented approach. We would > add > > > our > > > > >>> REST service, Alerts UI, and Management UI as services in Knox. > > > > >> Everything > > > > >>> would continue to function as it currently does but with all > > > > >> communication > > > > >>> going through Knox. LDAP authentication would be required when > > using > > > > >> Knox > > > > >>> and Knox will authenticate with the REST service by passing along > > an > > > > >>> Authorization header. Enabling Knox would be a manual process > that > > > > >>> involves deploying assets (Knox descriptor files) and changing > > > > >>> configuration. There would be no change to how the UI functions > by > > > > >> default > > > > >>> (without Knox) and either LDAP or JDBC authentication could still > > be > > > > >> used.. > > > > >>> > > > > >>> The third task involves enabling SSO with Knox. We would update > the > > > > REST > > > > >>> service so that it can authenticate with a Knox SSO token. We > would > > > > >>> provide documentation on how to update the Knox settings in > Ambari > > to > > > > >>> enable SSO. The Alerts/Management UI would need to expose > > > configuration > > > > >>> properties for the REST url and login url since these would be > > > > different > > > > >>> when Knox is enabled. We would also need to provide documentation > > on > > > > how > > > > >>> to make these UI configuration changes, based on the work done in > > > task > > > > > 1. > > > > >>> > > > > >>> An optional forth task would be exposing configuration settings > and > > > > >>> enabling Knox with Ambari. We would eliminate the manual steps > > > > necessary > > > > >>> for enabling Knox and instead automate those steps with an Ambari > > > input > > > > >>> control, similar to how LDAP is enabled. > > > > >>> > > > > >>> Thoughts on this plan? > > > > >>> > > > > >>> On Thu, Nov 15, 2018 at 10:22 AM James Sirota < > jsir...@apache.org> > > > > >> wrote: > > > > >>> > > > > >>>> In my view Knox SSO is such a minor feature when it comes to > > > Metron's > > > > >>>> capabilities than it's not worth supporting multiple scenarios > > where > > > > > it > > > > >>>> works with Knox or without Knox. Where we should be configurable > > > (and > > > > >>> are > > > > >>>> configurable) is on the analytics and stream processing. But > this? > > > As > > > > >>>> long as the UI authenticates securely I don't think anyone is > > going > > > to > > > > >>> care > > > > >>>> what proxy it's using. The code itself should be written in a > way > > > that > > > > >>>> it's pluggable so if we ever wanted to use another proxy or > > disable > > > it > > > > >>> all > > > > >>>> together we could. But this should not be a configuration we > pass > > on > > > > >> to > > > > >>>> the user. The added complexity is simply not worth it here. We > > have > > > > >> to > > > > >>>> start being opinionated about making sensible choices on behalf > of > > > the > > > > >>>> user. A sensible choice here is to run with Knox and LDAP. The > > JDBC > > > > >>>> component should exist for another release to allow the > community > > to > > > > >>>> migrate over to LDAP and then be deprecated. The code should > still > > > be > > > > >>>> pluggable and if anyone wanted to extend it to work with JDBC > they > > > > >> could, > > > > >>>> or if people wanted to plug in another proxy they could, but > this > > is > > > > >> not > > > > >>>> something we would officially support. > > > > >>>> > > > > >>>> Thanks, > > > > >>>> James > > > > >>>> > > > > >>>> 12.11.2018, 07:36, "Ryan Merriman" <merrim...@gmail.com>: > > > > >>>>> Let me clarify on exposing both legacy and Knox URLs at the > same > > > > >> time. > > > > >>>> The > > > > >>>>> base urls will look something like this: > > > > >>>>> > > > > >>>>> Legacy REST - http://node1:8082/api/v1 > > > > >>>>> Legacy Alerts UI - http://node1:4201:/alerts-list > > > > >>>>> > > > > >>>>> Knox REST - https://node1:8443/gateway/default/metron/api/v1 > > > > >>>>> Knox Alerts UI - > > > > >>>>> > https://node1:8443/gateway/default/metron-alerts-ui/alerts-list > > > > >>>>> > > > > >>>>> If Knox were turned on and the alerts UI deployed as is, it > would > > > > > not > > > > >>>>> work. This is because static assets are referenced with > > > > >>>>> http://node1:4201/assets/some-asset.js which does not include > > the > > > > >>>> correct > > > > >>>>> context path to the alerts UI in knox. To make it work, you > have > > to > > > > >> set > > > > >>>>> the base ref to "/gateway/default/metron-alerts-ui" so that > > static > > > > >>> assets > > > > >>>>> are referenced at > > > > >>>>> > > > > >>> > > > > > > https://node1:8443/gateway/default/metron-alerts-ui/assets/some-asset.js > > > > >>>> . > > > > >>>>> When you do that, the legacy alerts UI will no longer work. I > > guess > > > > >> the > > > > >>>>> point I'm trying to make is that we would have to switch > between > > > > > them > > > > >>> or > > > > >>>>> have 2 separate application running. I imagine most users only > > need > > > > >> one > > > > >>>> or > > > > >>>>> the other running so probably not an issue. > > > > >>>>> > > > > >>>>> Jon, the primary upgrade consideration I see is with > > > authentication. > > > > >> To > > > > >>>> be > > > > >>>>> able to use Knox, you would have to upgrade to LDAP-based > > > > >>> authentication > > > > >>>> if > > > > >>>>> you were still using JDBC-based authentication in REST. The > urls > > > > >> would > > > > >>>>> also change obviously. > > > > >>>>> > > > > >>>>> On Sun, Nov 11, 2018 at 6:38 PM zeo...@gmail.com < > > zeo...@gmail.com > > > > > > > > >>>> wrote: > > > > >>>>> > > > > >>>>>> Phew, that was quite the thread to catch up on. > > > > >>>>>> > > > > >>>>>> I agree that this should be optional/pluggable to start, and > I'm > > > > >>>> interested > > > > >>>>>> to hear the issues as they relate to upgrading an existing > > cluster > > > > >>>> (given > > > > >>>>>> the suggested approach) and exposing both legacy and knox URLs > > at > > > > >> the > > > > >>>> same > > > > >>>>>> time. > > > > >>>>>> > > > > >>>>>> Jon > > > > >>>>>> > > > > >>>>>> On Fri, Nov 9, 2018, 4:46 PM Michael Miklavcic < > > > > >>>>>> michael.miklav...@gmail.com> > > > > >>>>>> wrote: > > > > >>>>>> > > > > >>>>>>> A couple more things, and I think this goes without saying - > > > > >>>> whatever we > > > > >>>>>> do > > > > >>>>>>> with Knox should NOT > > > > >>>>>>> > > > > >>>>>>> 1. Require unit and integration tests to use Knox > > > > >>>>>>> 2. Break fulldev > > > > >>>>>>> > > > > >>>>>>> Also, I don't know that I saw you mention this, but I'm > unsure > > > > >> how > > > > >>> we > > > > >>>>>>> should leverage Knox as a core piece of the platform. i.e. > > should > > > > >>> we > > > > >>>> make > > > > >>>>>>> this required or optional? I'm open to hearing opinions on > > this, > > > > >>> but > > > > >>>> I'm > > > > >>>>>>> inclined to keep this a pluggable option. > > > > >>>>>>> > > > > >>>>>>> Mike > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> On Fri, Nov 9, 2018 at 2:42 PM Michael Miklavcic < > > > > >>>>>>> michael.miklav...@gmail.com> wrote: > > > > >>>>>>> > > > > >>>>>>>> Thanks for the update Ryan. Per my earlier comments, I > thought > > > > >> it > > > > >>>> might > > > > >>>>>>> be > > > > >>>>>>>> the case that we could dramatically simplify this by > > leveraging > > > > >>>> Knox's > > > > >>>>>>>> proxy capabilities, and per your research that appears to be > > > > >> the > > > > >>>> case. > > > > >>>>>>> This > > > > >>>>>>>> is a dramatic simplification and improvement of this feature > > > > >> imo, > > > > >>>> +1. > > > > >>>>>> I'm > > > > >>>>>>>> also +1 on a couple distinct steps that you've laid out: fix > > > > >> the > > > > >>> UI > > > > >>>>>>> issues > > > > >>>>>>>> in master, then add Knox for SSO. That should help mitigate > > > > >>> issues > > > > >>>> with > > > > >>>>>>>> merge conflicts with ongoing development. > > > > >>>>>>>> > > > > >>>>>>>>> I think it will be a challenge exposing the UIs through > both > > > > >>> the > > > > >>>> Knox > > > > >>>>>>>> url and legacy urls at the same time. > > > > >>>>>>>> I'm not sure I understand the issue here. Are you referring > to > > > > >>> this > > > > >>>>>>>> comment? "Added a ng build option to build the UI with base > > > > >> href > > > > >>>> set to > > > > >>>>>>>> Knox base path." Isn't it just a matter of URL > > > > >>>> rewriting/forwarding? I > > > > >>>>>>>> thought we'd be exposing the URL's directly in one context, > > and > > > > >>>> through > > > > >>>>>>>> Knox in the other. Either way, it seems like we should be > able > > > > >> to > > > > >>>>>>> provide a > > > > >>>>>>>> dynamic base path through configuration in our web > > > > >> applications. > > > > >>>> I'd > > > > >>>>>>> expect > > > > >>>>>>>> to modify that property based on whether Knox is configured > or > > > > >>> not. > > > > >>>>>>>> > > > > >>>>>>>>> I'm also not clear on how one would use Knox with REST set > to > > > > >>>> legacy > > > > >>>>>>>> JDBC-based authentication. As far as I know Knox does not > > > > >> support > > > > >>>> JDBC > > > > >>>>>> so > > > > >>>>>>>> there would be a mismatch between Knox and REST. > > > > >>>>>>>> I'm OK with not having Knox work with JDBC. That's a feature > > of > > > > >>>> Knox > > > > >>>>>> and > > > > >>>>>>>> probably not something we care much about. > > > > >>>>>>>> > > > > >>>>>>>>> We could initially make Knox an optional feature that > > requires > > > > >>>> setup > > > > >>>>>>> with > > > > >>>>>>>> the help of some documentation (like Kerberos) while keeping > > > > >> the > > > > >>>> system > > > > >>>>>>> the > > > > >>>>>>>> way it is now by default. > > > > >>>>>>>> Sounds good to me. > > > > >>>>>>>> > > > > >>>>>>>>> I imagine we'll deprecate JDBC-based authentication at some > > > > >>>> point so > > > > >>>>>>>> that may be a good time to switch. > > > > >>>>>>>> I would like to announce deprecation in our next release and > > > > >> move > > > > >>>> to > > > > >>>>>>>> remove it in a following release. > > > > >>>>>>>> > > > > >>>>>>>> Thanks for taking this on and great job laying things out. > > > > >>>>>>>> > > > > >>>>>>>> Thanks, > > > > >>>>>>>> Mike > > > > >>>>>>>> > > > > >>>>>>>> On Fri, Nov 9, 2018 at 2:09 PM Ryan Merriman < > > > > >>> merrim...@gmail.com> > > > > >>>>>>> wrote: > > > > >>>>>>>> > > > > >>>>>>>>> I have spent some time recently reviewing this discussion > and > > > > >>> the > > > > >>>>>>> feature > > > > >>>>>>>>> branch that Simon put out. I think this is an important > > > > >> feature > > > > >>>> and > > > > >>>>>>> want > > > > >>>>>>>>> to move it forward. I started another discussion on adding > > > > >> Knox > > > > >>> to > > > > >>>>>> our > > > > >>>>>>>>> stack but this discussion has a lot of good context so I > will > > > > >>>> continue > > > > >>>>>>> it > > > > >>>>>>>>> here. > > > > >>>>>>>>> > > > > >>>>>>>>> I think the main point of contention was that this feature > > > > >>> branch > > > > >>>>>>> included > > > > >>>>>>>>> several different architectural changes and it was unclear > if > > > > >>> they > > > > >>>>>> were > > > > >>>>>>>>> needed and if so, could be done separately. Fortunately > LDAP > > > > >>>>>>>>> authentication has been accepted into master so we can > cross > > > > >> it > > > > >>>> off > > > > >>>>>> the > > > > >>>>>>>>> list. From my understanding of the points people have made, > > > > >> that > > > > >>>>>> leaves > > > > >>>>>>>>> Knox related SSO changes and migrating expressjs to a > > > > >> different, > > > > >>>>>>> JVM-based > > > > >>>>>>>>> web server that includes proxying capabilities (Zuul). > > > > >>>>>>>>> > > > > >>>>>>>>> I think everyone agrees that if we can limit the scope to > > just > > > > >>>> Knox > > > > >>>>>>>>> related > > > > >>>>>>>>> SSO changes that would be ideal. I believe I have found a > way > > > > >> to > > > > >>>> do > > > > >>>>>>> that > > > > >>>>>>>>> while working on a small POC this week. The key to this > > (Simon > > > > >>>>>> alluded > > > > >>>>>>> to > > > > >>>>>>>>> it earlier) is to put both REST and our UIs behind Knox. I > > > > >>>> initially > > > > >>>>>>> was > > > > >>>>>>>>> focused on just adding REST as a service in Knox and > decided > > > > >> to > > > > >>>>>>> experiment > > > > >>>>>>>>> with also adding our UIs. After I did this it became clear > > > > >> that > > > > >>>> this > > > > >>>>>>>>> simplifies things considerably: > > > > >>>>>>>>> > > > > >>>>>>>>> - The REST app and the UIs are now served from the same > host > > > > >> so > > > > >>>>>> CORS > > > > >>>>>>>>> concerns go away. > > > > >>>>>>>>> - We no longer need to worry about proxying REST requests > > from > > > > >>> the > > > > >>>>>>> UIs > > > > >>>>>>>>> with express or Zuul because Knox handles that for us. This > > > > >> will > > > > >>>>>>> make > > > > >>>>>>>>> our > > > > >>>>>>>>> express configuration even simpler. In fact, all we need > is a > > > > >>>>>> simple > > > > >>>>>>>>> way > > > > >>>>>>>>> to serve static UI assets. > > > > >>>>>>>>> - We no longer need to check for SSO tokens and redirect in > > > > >> the > > > > >>> UI > > > > >>>>>>>>> web/app servers (or the REST app for that matter) because > > Knox > > > > >>>>>>> handles > > > > >>>>>>>>> that > > > > >>>>>>>>> for us. > > > > >>>>>>>>> - The UIs can now easily access any Knox service (not just > > our > > > > >>>> REST > > > > >>>>>>>>> app) > > > > >>>>>>>>> without any extra proxy configuration. > > > > >>>>>>>>> - SSO token authentication is only necessary in REST so > there > > > > >> is > > > > >>>> no > > > > >>>>>>>>> need > > > > >>>>>>>>> to create shared Spring modules or split functionality out. > > > > >>>>>>>>> > > > > >>>>>>>>> The most significant change I had to make (borrowed from > > > > >> Simon's > > > > >>>>>> feature > > > > >>>>>>>>> branch) was the SSO token authentication mentioned above. > The > > > > >>>> primary > > > > >>>>>>>>> short term benefit with this approach is that outside of > some > > > > >>>> general > > > > >>>>>>>>> deficiencies unrelated to this our UI architecture doesn't > > > > >> need > > > > >>> to > > > > >>>>>>>>> fundamentally change. I could summarize the changes as: > > > > >>>>>>>>> > > > > >>>>>>>>> - Knox install and configuration (setting up REST and the > > > > >> alerts > > > > >>>> UI > > > > >>>>>>> as > > > > >>>>>>>>> Knox services) > > > > >>>>>>>>> - Added Knox SSO token authentication to REST > > > > >>>>>>>>> - Updated REST urls in the UI code (should be configurable) > > > > >>>>>>>>> - Fixed a few UI bugs where relative paths were not being > > used > > > > >>>>>>>>> - Added a ng build option to build the UI with base href > set > > > > >> to > > > > >>>>>> Knox > > > > >>>>>>>>> base path ( > > > > >>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://github.com/angular/angular-cli/wiki/build#base-tag-handling-in-indexhtml > > > > >>>>>>>>> ) > > > > >>>>>>>>> > > > > >>>>>>>>> Most the UI changes are preexisting, minor issues that > could > > > > >> be > > > > >>>> fixed > > > > >>>>>>>>> directly in master. We would need to think of an approach > for > > > > >>> the > > > > >>>>>> base > > > > >>>>>>>>> href build requirement but I'm sure it's not that bad. > > > > >>>>>>>>> > > > > >>>>>>>>> However there will be some backwards compatibility issues > we > > > > >>> would > > > > >>>>>> need > > > > >>>>>>> to > > > > >>>>>>>>> think through. I think it will be a challenge exposing the > > UIs > > > > >>>>>> through > > > > >>>>>>>>> both the Knox url and legacy urls at the same time. I'm > also > > > > >> not > > > > >>>>>> clear > > > > >>>>>>> on > > > > >>>>>>>>> how one would use Knox with REST set to legacy JDBC-based > > > > >>>>>>> authentication. > > > > >>>>>>>>> As far as I know Knox does not support JDBC so there would > be > > > > >> a > > > > >>>>>> mismatch > > > > >>>>>>>>> between Knox and REST. Knox does have the ability to pass > > > > >> along > > > > >>>> basic > > > > >>>>>>>>> authentication headers so LDAP in REST would work. We could > > > > >>>> initially > > > > >>>>>>>>> make > > > > >>>>>>>>> Knox an optional feature that requires setup with the help > of > > > > >>> some > > > > >>>>>>>>> documentation (like Kerberos) while keeping the system the > > way > > > > >>> it > > > > >>>> is > > > > >>>>>> now > > > > >>>>>>>>> by > > > > >>>>>>>>> default. I imagine we'll deprecate JDBC-based > authentication > > > > >> at > > > > >>>> some > > > > >>>>>>>>> point > > > > >>>>>>>>> so that may be a good time to switch. > > > > >>>>>>>>> > > > > >>>>>>>>> What do people think about this approach? Concerns? Are > there > > > > >>> any > > > > >>>>>> huge > > > > >>>>>>>>> holes in this I'm not thinking about? > > > > >>>>>>>>> > > > > >>>>>>>>> I want to highlight that the work Simon did in his feature > > > > >>> branch > > > > >>>> was > > > > >>>>>>>>> crucial to better understanding this. I am pretty sure > we'll > > > > >> end > > > > >>>> up > > > > >>>>>>>>> reusing a lot code from that branch. > > > > >>>>>>>>> > > > > >>>>>>>>> On Thu, Sep 27, 2018 at 6:30 PM Michael Miklavcic < > > > > >>>>>>>>> michael.miklav...@gmail.com> wrote: > > > > >>>>>>>>> > > > > >>>>>>>>>> Apparently, I hit send on my last email before finishing > my > > > > >>>> synopsis > > > > >>>>>>>>> (per > > > > >>>>>>>>>> @Otto's Q in Slack). To summarize, based on my current > > > > >>>>>> understanding I > > > > >>>>>>>>>> believe that each of the feature branch changes I've > outline > > > > >>>> above > > > > >>>>>> are > > > > >>>>>>>>>> units of work that are related, yet should be executed on > > > > >>>>>>> independently. > > > > >>>>>>>>>> Knox SSO in its own feature branch. Migrating technologies > > > > >>> like > > > > >>>>>> NodeJs > > > > >>>>>>>>> or > > > > >>>>>>>>>> migrating the auth DB to LDAP seem like they belong in > their > > > > >>> own > > > > >>>>>>>>> separate > > > > >>>>>>>>>> PR's or feature branches. > > > > >>>>>>>>>> > > > > >>>>>>>>>> Thanks, > > > > >>>>>>>>>> Mike > > > > >>>>>>>>>> > > > > >>>>>>>>>> On Thu, Sep 27, 2018 at 4:08 PM Casey Stella < > > > > >>>> ceste...@gmail.com> > > > > >>>>>>>>> wrote: > > > > >>>>>>>>>> > > > > >>>>>>>>>>> I'm coming in late to the game here, but for my mind a > > > > >>> feature > > > > >>>>>>> branch > > > > >>>>>>>>>>> should involve the minimum architectural change to > > > > >>> accomplish > > > > >>>> a > > > > >>>>>>> given > > > > >>>>>>>>>>> feature. > > > > >>>>>>>>>>> The feature in question is SSO integration. It seems to > me > > > > >>>> that > > > > >>>>>> the > > > > >>>>>>>>>>> operative question is can we do the feature without > making > > > > >>> the > > > > >>>>>> OTHER > > > > >>>>>>>>>>> architectural change > > > > >>>>>>>>>>> (e.g. migrating from expressjs to spring boot + zuul). I > > > > >>> would > > > > >>>>>>> argue > > > > >>>>>>>>>> that > > > > >>>>>>>>>>> if we WANT to do that, then it should be a separate > > > > >> feature > > > > >>>>>> branch. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> Thus, I leave with a question: is there a way to > > > > >> accomplish > > > > >>>> this > > > > >>>>>>>>> feature > > > > >>>>>>>>>>> without ripping out expressjs? > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> - If so and it is feasible, I would argue that we should > > > > >>>>>> decouple > > > > >>>>>>>>> this > > > > >>>>>>>>>>> into a separate feature branch. > > > > >>>>>>>>>>> - If so and it is infeasible, I'd like to hear an > argument > > > > >>> as > > > > >>>>>> to > > > > >>>>>>>>> the > > > > >>>>>>>>>>> infeasibility and let's decide given that > > > > >>>>>>>>>>> - If it is not possible, then I'd argue that we should > > > > >> keep > > > > >>>>>> them > > > > >>>>>>>>>> coupled > > > > >>>>>>>>>>> and move this through as-is. > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On a side-note, it feels a bit weird that we're narrowing > > > > >>> to a > > > > >>>>>>> bundled > > > > >>>>>>>>>>> proxy, rather than having that be a pluggable thing. I'm > > > > >> not > > > > >>>>>> super > > > > >>>>>>>>>>> knowledgeable in this space, so I apologize > > > > >>>>>>>>>>> in advance if this is naive, but isn't this a pluggable, > > > > >>>> external > > > > >>>>>>>>>> component > > > > >>>>>>>>>>> (e.g. nginx)? > > > > >>>>>>>>>>> > > > > >>>>>>>>>>> On Thu, Sep 27, 2018 at 5:05 PM Michael Miklavcic < > > > > >>>>>>>>>>> michael.miklav...@gmail.com> wrote: > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>> I've spent some more time reading through Simon's > > > > >> response > > > > >>>> and > > > > >>>>>> the > > > > >>>>>>>>>> added > > > > >>>>>>>>>>>> sequence diagram. This is definitely helpful - thank you > > > > >>>> Simon. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> I need to redact my initial list: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> 1. Node migrated to Spring Boot, expressjs migrated to a > > > > >>>>>>>>>>>> non-JS/non-NodeJs proxying mechanism (ie Zuul in this > > > > >>> case) > > > > >>>>>>>>>>>> 2. JDBC removed completely in favor of LDAP > > > > >>>>>>>>>>>> 3. Knox/SSO > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> I'm a bit conflicted on the best way to move forward and > > > > >>>> would > > > > >>>>>>> like > > > > >>>>>>>>>> some > > > > >>>>>>>>>>>> thoughts from other community members on this. I think > > > > >> an > > > > >>>>>> argument > > > > >>>>>>>>> can > > > > >>>>>>>>>> be > > > > >>>>>>>>>>>> made that 1 and 2 are independent of 3, and should/could > > > > >>>> really > > > > >>>>>> be > > > > >>>>>>>>>>>> independent PR's against master. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> The need for a replacement for expressjs (Zuul in this > > > > >>>> case) is > > > > >>>>>> an > > > > >>>>>>>>>>> artifact > > > > >>>>>>>>>>>> that our request/response cycle for REST calls is a > > > > >> simple > > > > >>>>>> matter > > > > >>>>>>> of > > > > >>>>>>>>>>>> forwarding with some additional headers for > > > > >>> authentication. > > > > >>>>>>> There's > > > > >>>>>>>>> a > > > > >>>>>>>>>>>> JSESSIONID managed by the client browser in our current > > > > >>>>>>>>> architecture, > > > > >>>>>>>>>> for > > > > >>>>>>>>>>>> example. You login to the alerts or the management UI > > > > >>> which > > > > >>>>>>>>> forwards a > > > > >>>>>>>>>>>> request to REST, which looks up credentials in a backend > > > > >>>>>> database, > > > > >>>>>>>>> and > > > > >>>>>>>>>>>> passes the results back up the chain. All browser > > > > >> requests > > > > >>>> go > > > > >>>>>>>>> directly > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>> the specific UI you're working with - this is the CORS > > > > >>>> problem. > > > > >>>>>>> You > > > > >>>>>>>>>>> can't, > > > > >>>>>>>>>>>> without some effort with headers for adding other > > > > >> domains > > > > >>>> to the > > > > >>>>>>>>> safe > > > > >>>>>>>>>>> list > > > > >>>>>>>>>>>> or disabling the security check for CORS, make remote > > > > >>> calls > > > > >>>>>>>>> directly to > > > > >>>>>>>>>>>> REST. That's why we proxy. Switching over to Spring Boot > > > > >>>> leaves > > > > >>>>>> a > > > > >>>>>>>>> gap > > > > >>>>>>>>>>> with > > > > >>>>>>>>>>>> expressjs having handled the proxying and filtering, > > > > >> since > > > > >>>> it's > > > > >>>>>>> only > > > > >>>>>>>>>>>> available to a NodeJs application (it's server-side > > > > >>>> javascript > > > > >>>>>> vs > > > > >>>>>>>>> the > > > > >>>>>>>>>>>> client side javascript deployed via our Angular > > > > >>>> applications). > > > > >>>>>>> Enter > > > > >>>>>>>>>>> Zuul, > > > > >>>>>>>>>>>> which now effectively handles that. At runtime, Zuul is > > > > >> a > > > > >>>> part > > > > >>>>>> of > > > > >>>>>>>>> the > > > > >>>>>>>>>>>> Spring app that serves up our UI's. It handles the > > > > >>> requests > > > > >>>> via > > > > >>>>>>>>>>> filtering, > > > > >>>>>>>>>>>> forwards them to REST, manages the response back to the > > > > >>>> client. > > > > >>>>>>> Very > > > > >>>>>>>>>>>> similar to what expressjs was doing, per my current > > > > >>>>>> understanding. > > > > >>>>>>>>> The > > > > >>>>>>>>>>>> sequence diagrams Simon added are useful, and I think > > > > >> some > > > > >>>> of > > > > >>>>>> what > > > > >>>>>>>>> was > > > > >>>>>>>>>>> less > > > > >>>>>>>>>>>> clear was what we currently vs what the new changes are > > > > >>>> doing to > > > > >>>>>>> the > > > > >>>>>>>>>>>> architecture. This is no fault of Simon's - there simply > > > > >>>> wasn't > > > > >>>>>>> any > > > > >>>>>>>>>>>> architecture diagrams/documents around this before. > > > > >> Here's > > > > >>>> my > > > > >>>>>>>>>> impression > > > > >>>>>>>>>>> of > > > > >>>>>>>>>>>> the very very basic current state - someone more > > > > >> familiar > > > > >>>> with > > > > >>>>>>> this > > > > >>>>>>>>>>>> architecture please advise if I'm incorrect about > > > > >> anything > > > > >>>>>>> (probably > > > > >>>>>>>>>>> Ryan). > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> https://imgur.com/f8GtSmh > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Zuul would be replacing the bit about expressjs in the > > > > >>>> diagram, > > > > >>>>>>> and > > > > >>>>>>>>>>> instead > > > > >>>>>>>>>>>> of node we have spring boot. This covers 1. 2 and 3 are > > > > >>>> other > > > > >>>>>>>>> issues. > > > > >>>>>>>>>> I'd > > > > >>>>>>>>>>>> like to see similar exposition of those server processes > > > > >>>> with > > > > >>>>>> knox > > > > >>>>>>>>>>>> involved. I imagine in that case we bump up from 3 to 4 > > > > >>>> server > > > > >>>>>>>>>> instances > > > > >>>>>>>>>>>> for the additional knox endpoint. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> Mike > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> On Wed, Sep 19, 2018 at 11:28 AM James Sirota < > > > > >>>>>> jsir...@apache.org > > > > >>>>>>>> > > > > >>>>>>>>>>> wrote: > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>>> Thank you, Simon. The diagrams help a lot > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> 19.09.2018, 21:27, "Simon Elliston Ball" < > > > > >>>>>>>>>> si...@simonellistonball.com > > > > >>>>>>>>>>>> : > > > > >>>>>>>>>>>>>> To clarify some of this I've put some documentation > > > > >>> into > > > > >>>>>>>>>>>>>> https://github.com/apache/metron/pull/1203 under > > > > >>>>>> METRON-1755 > > > > >>>>>>> ( > > > > >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/METRON-1755). > > > > >>>>>> Hopefully > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>> diagrams > > > > >>>>>>>>>>>>>> there should make it clearer. > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> Simon > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> On Tue, 18 Sep 2018 at 14:17, Simon Elliston Ball < > > > > >>>>>>>>>>>>>> si...@simonellistonball.com> wrote: > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Hi Mike, > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Some good points here which could do with some > > > > >>>>>>> clarification. > > > > >>>>>>>>> I > > > > >>>>>>>>>>>> suspect > > > > >>>>>>>>>>>>>>> the architecture documentation could be clearer and > > > > >>>> fill > > > > >>>>>> in > > > > >>>>>>>>> some > > > > >>>>>>>>>> of > > > > >>>>>>>>>>>>> these > > > > >>>>>>>>>>>>>>> gaps, and I'll have a look at working on that and > > > > >>>>>> providing > > > > >>>>>>>>> some > > > > >>>>>>>>>>>>> diagrams. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> The short version is that the Zuul proxy gateway > > > > >> has > > > > >>>> been > > > > >>>>>>>>> added > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>>> replace > > > > >>>>>>>>>>>>>>> the Nodejs express proxy used to gateway the REST > > > > >> api > > > > >>>>>> calls > > > > >>>>>>> in > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>> current > > > > >>>>>>>>>>>>>>> hosts. This is done in both cases to avoid CORS > > > > >>>>>> restrictions > > > > >>>>>>>>> by > > > > >>>>>>>>>>>>> allowing > > > > >>>>>>>>>>>>>>> the same host that serves the UI files to proxy > > > > >> call > > > > >>> to > > > > >>>>>> the > > > > >>>>>>>>> API. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> The choice of Zuul was partly a pragmatic one (it's > > > > >>> the > > > > >>>>>> one > > > > >>>>>>>>>> that's > > > > >>>>>>>>>>>>> there > > > > >>>>>>>>>>>>>>> in the box as it were with Spring Boot, which we > > > > >> use > > > > >>>> for > > > > >>>>>> the > > > > >>>>>>>>> REST > > > > >>>>>>>>>>>> API, > > > > >>>>>>>>>>>>> via > > > > >>>>>>>>>>>>>>> the Spring Cloud Netflix project which wraps a > > > > >> bunch > > > > >>> of > > > > >>>>>>>>> related > > > > >>>>>>>>>>>> pieces > > > > >>>>>>>>>>>>> into > > > > >>>>>>>>>>>>>>> Spring). The choice of Spring Boot to host the UIs > > > > >>>>>>> themselves > > > > >>>>>>>>> was > > > > >>>>>>>>>>>>> similarly > > > > >>>>>>>>>>>>>>> for parity with the REST host, to simplify the > > > > >> stack > > > > >>>> (we > > > > >>>>>>>>> remove > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>>> occasionally problematic need to install nodejs on > > > > >>>> target > > > > >>>>>>>>>> servers, > > > > >>>>>>>>>>>>> which is > > > > >>>>>>>>>>>>>>> outside of the regular OS and HDP stacks we > > > > >> support). > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Arguably, the Zuul proxy is not necessary if we > > > > >> force > > > > >>>>>>>>> everything > > > > >>>>>>>>>>>>> through a > > > > >>>>>>>>>>>>>>> Knox instance, since Knox would provide a single > > > > >>>> endpoint. > > > > >>>>>>> We > > > > >>>>>>>>>>>> probably > > > > >>>>>>>>>>>>>>> however don't want to force Knox and SSL, hence > > > > >> using > > > > >>>> Zuul > > > > >>>>>>> to > > > > >>>>>>>>>> keep > > > > >>>>>>>>>>> it > > > > >>>>>>>>>>>>>>> closer to our current architecture. Zuul does some > > > > >>>> other > > > > >>>>>>> nice > > > > >>>>>>>>>>> things, > > > > >>>>>>>>>>>>> which > > > > >>>>>>>>>>>>>>> might help us in future, so it's really about > > > > >> laying > > > > >>>> down > > > > >>>>>>> some > > > > >>>>>>>>>>>> options > > > > >>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>> potentially doing micro-services style things at a > > > > >>>> later > > > > >>>>>>> date. > > > > >>>>>>>>>> I'm > > > > >>>>>>>>>>>> not > > > > >>>>>>>>>>>>>>> saying we have to, or even should go that way, it > > > > >>> will > > > > >>>>>> just > > > > >>>>>>>>> make > > > > >>>>>>>>>>> life > > > > >>>>>>>>>>>>>>> easier later if we decide to. It will also help us > > > > >> if > > > > >>>> we > > > > >>>>>>> want > > > > >>>>>>>>> to > > > > >>>>>>>>>>> add > > > > >>>>>>>>>>>>> HA, > > > > >>>>>>>>>>>>>>> circuit breaking etc to the architecture at a later > > > > >>>> date. > > > > >>>>>>> That > > > > >>>>>>>>>>> said, > > > > >>>>>>>>>>>> I > > > > >>>>>>>>>>>>>>> regret that I ever said the word micro-services, > > > > >>> since > > > > >>>>>> it's > > > > >>>>>>>>>> caused > > > > >>>>>>>>>>>>>>> confusion. Just think of it as a proxy to deal with > > > > >>> the > > > > >>>>>> CORS > > > > >>>>>>>>>>> problem. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Zuul is implemented as a set of filters, but we are > > > > >>> not > > > > >>>>>>> using > > > > >>>>>>>>> it > > > > >>>>>>>>>>> for > > > > >>>>>>>>>>>>> its > > > > >>>>>>>>>>>>>>> authentication filtering. We're using it as a > > > > >> proxy. > > > > >>>> Shiro > > > > >>>>>>> is > > > > >>>>>>>>> an > > > > >>>>>>>>>>>>>>> authentication framework, and could arguably used > > > > >> to > > > > >>>>>> provide > > > > >>>>>>>>> the > > > > >>>>>>>>>>>>> security > > > > >>>>>>>>>>>>>>> piece, but frankly wrapping shiro as a replacement > > > > >>> for > > > > >>>>>>> Spring > > > > >>>>>>>>>>>> Security > > > > >>>>>>>>>>>>> in a > > > > >>>>>>>>>>>>>>> Spring application seemed like it will make life a > > > > >>> lot > > > > >>>>>>> harder. > > > > >>>>>>>>>> This > > > > >>>>>>>>>>>>> could > > > > >>>>>>>>>>>>>>> be done, but it's not the native happy path, and > > > > >>> would > > > > >>>>>> pull > > > > >>>>>>> in > > > > >>>>>>>>>>>>> additional > > > > >>>>>>>>>>>>>>> dependencies that duplicate functionality that's > > > > >>>> already > > > > >>>>>>>>> embedded > > > > >>>>>>>>>>> in > > > > >>>>>>>>>>>>> Spring > > > > >>>>>>>>>>>>>>> Security. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> The version of Knox used is the default from HDP. > > > > >> The > > > > >>>> link > > > > >>>>>>>>>> version > > > > >>>>>>>>>>>> you > > > > >>>>>>>>>>>>>>> mention is a docs link. I'll update it to be the > > > > >>> older > > > > >>>>>>>>> version, > > > > >>>>>>>>>>> which > > > > >>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>> the same and we can decide if we want to maintain > > > > >> the > > > > >>>>>>>>> freshness > > > > >>>>>>>>>> of > > > > >>>>>>>>>>> it > > > > >>>>>>>>>>>>> when > > > > >>>>>>>>>>>>>>> we look to upgrade underlying patterns. Either way, > > > > >>> the > > > > >>>>>>>>> content > > > > >>>>>>>>>> is > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>> same. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I did consider other hosting mechanisms, including > > > > >>>>>> Undertow > > > > >>>>>>> a > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> If you have a different suggestion to using the > > > > >>> Spring > > > > >>>>>>> default > > > > >>>>>>>>>> ways > > > > >>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>> doing things, or we want to use a framework other > > > > >>> than > > > > >>>>>>> Spring > > > > >>>>>>>>> for > > > > >>>>>>>>>>>> this, > > > > >>>>>>>>>>>>>>> then maybe we could change to that, but the route > > > > >>>> chosen > > > > >>>>>>> here > > > > >>>>>>>>> is > > > > >>>>>>>>>>>>> definitely > > > > >>>>>>>>>>>>>>> the easy path in the context of the decision made > > > > >> to > > > > >>>> use > > > > >>>>>>>>> Spring > > > > >>>>>>>>>> in > > > > >>>>>>>>>>>>> metron > > > > >>>>>>>>>>>>>>> rest, and if anything opens up our choices while > > > > >>>>>> minimising, > > > > >>>>>>>>> in > > > > >>>>>>>>>>> fact > > > > >>>>>>>>>>>>>>> reducing, our dependency management overhead. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> I hope that explains some of the thinking behind > > > > >> the > > > > >>>>>> choices > > > > >>>>>>>>>> made, > > > > >>>>>>>>>>>> but > > > > >>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>> guiding principals I followed were: > > > > >>>>>>>>>>>>>>> * Don't fight the framework if you don't have to > > > > >>>>>>>>>>>>>>> * Reduce the need for additional installation > > > > >> pieces > > > > >>>> and > > > > >>>>>>> third > > > > >>>>>>>>>>> party > > > > >>>>>>>>>>>>> repos > > > > >>>>>>>>>>>>>>> * Minimize dependencies we would have to manage > > > > >>>>>>>>>>>>>>> * Avoid excessive change of the architecture, or > > > > >>>> forcing > > > > >>>>>>>>> users to > > > > >>>>>>>>>>>> adopt > > > > >>>>>>>>>>>>>>> Knox if they didn't want the SSL overhead. > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> Simon > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> On Tue, 18 Sep 2018 at 02:46, Michael Miklavcic < > > > > >>>>>>>>>>>>>>> michael.miklav...@gmail.com> wrote: > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Thanks for the write-up Ryan, this is a great > > > > >>> start. I > > > > >>>>>> have > > > > >>>>>>>>> some > > > > >>>>>>>>>>>>> further > > > > >>>>>>>>>>>>>>>> questions based on your feedback and in addition > > > > >> to > > > > >>> my > > > > >>>>>>>>> initial > > > > >>>>>>>>>>>> thread. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Just for clarification, what version of Knox are > > > > >> we > > > > >>>>>> using? > > > > >>>>>>>>> HDP > > > > >>>>>>>>>>>> 2.6.5, > > > > >>>>>>>>>>>>>>>> which > > > > >>>>>>>>>>>>>>>> is what we currently run full dev against, > > > > >> supports > > > > >>>>>> 0.12.0. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_release-notes/content/comp_versions.html > > > > >>>>>>>>>>>>>>>> . > > > > >>>>>>>>>>>>>>>> I see references to Knox 1.1.0 (latest) in this > > > > >>>> committed > > > > >>>>>>> PR > > > > >>>>>>>>> - > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://github.com/apache/metron/pull/1111/files#diff-70b412194819f3cb829566f05d77c1a6R122 > > > > >>>>>>>>>>>>>>>> . > > > > >>>>>>>>>>>>>>>> This is probably just a super small mismatch, and > > > > >> it > > > > >>>>>>> probably > > > > >>>>>>>>>> goes > > > > >>>>>>>>>>>>> without > > > > >>>>>>>>>>>>>>>> saying, but I just want to be doubly sure that > > > > >> we're > > > > >>>>>>>>> installing > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> default > > > > >>>>>>>>>>>>>>>> via the standard install mechanism as opposed to > > > > >>>>>> something > > > > >>>>>>>>>>> separate > > > > >>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>> manual. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On the subject of Zuul wrt Nodejs filters. I'd > > > > >> like > > > > >>> to > > > > >>>>>> hear > > > > >>>>>>>>> some > > > > >>>>>>>>>>>> more > > > > >>>>>>>>>>>>>>>> detail on: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> 1. Why do we need filtering via Zuul? For > > > > >> instance, > > > > >>> is > > > > >>>>>>>>>>> filtering > > > > >>>>>>>>>>>>>>>> routing > > > > >>>>>>>>>>>>>>>> not handled by Knox? From the beginner docs: "The > > > > >>>>>>> gateway > > > > >>>>>>>>>>> itself > > > > >>>>>>>>>>>>> is a > > > > >>>>>>>>>>>>>>>> layer > > > > >>>>>>>>>>>>>>>> over an embedded Jetty JEE server. At the very > > > > >>> highest > > > > >>>>>>>>> level > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> gateway > > > > >>>>>>>>>>>>>>>> processes requests by using request URLs to lookup > > > > >>>>>>>>> specific > > > > >>>>>>>>>> JEE > > > > >>>>>>>>>>>>> Servlet > > > > >>>>>>>>>>>>>>>> Filter chain that is used to process the request. > > > > >>> The > > > > >>>>>>>>> gateway > > > > >>>>>>>>>>>>> framework > > > > >>>>>>>>>>>>>>>> provides extensible mechanisms to assemble chains > > > > >> of > > > > >>>>>>>>> custom > > > > >>>>>>>>>>>> filters > > > > >>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>> support secured access to services." [1] > > > > >>>>>>>>>>>>>>>> 2. What other library options were considered for > > > > >>> this > > > > >>>>>>>>>> feature > > > > >>>>>>>>>>>> and > > > > >>>>>>>>>>>>> how > > > > >>>>>>>>>>>>>>>> was it chosen over the others? I search on "apache > > > > >>>>>>> spring > > > > >>>>>>>>> web > > > > >>>>>>>>>>>>> filters" > > > > >>>>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>> it's almost all about Shiro - > > > > >>>>>>>>>>>> https://shiro.apache.org/spring.html. > > > > >>>>>>>>>>>>> I > > > > >>>>>>>>>>>>>>>> also see quite a bit about filtering for Spring > > > > >> Boot > > > > >>>>>>>>>>> applications > > > > >>>>>>>>>>>>> along > > > > >>>>>>>>>>>>>>>> with a write-up of how to accomplish the same with > > > > >>> Web > > > > >>>>>>> MVC > > > > >>>>>>>>>>> here - > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://stackoverflow.com/questions/19825946/how-to-add-a-filter-class-in-spring-boot > > > > >>>>>>>>>>>>>>>> . > > > > >>>>>>>>>>>>>>>> The Knox documentation boilerplate examples are > > > > >> also > > > > >>>>>>> using > > > > >>>>>>>>>>> Shiro. > > > > >>>>>>>>>>>>>>>> "shiro.ini - The configuration file for the Shiro > > > > >>>>>>>>>>> authentication > > > > >>>>>>>>>>>>>>>> provider’s > > > > >>>>>>>>>>>>>>>> filters. This information is derived from the > > > > >>>>>>> information > > > > >>>>>>>>> in > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> provider > > > > >>>>>>>>>>>>>>>> section of the topology file." [1] > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> My assumption is that there are deliberate > > > > >> decisions > > > > >>>> in > > > > >>>>>>>>> favor of > > > > >>>>>>>>>>>> this > > > > >>>>>>>>>>>>> mix > > > > >>>>>>>>>>>>>>>> of technologies over others, and I think some > > > > >>>> additional > > > > >>>>>>>>>>> explanation > > > > >>>>>>>>>>>>> will > > > > >>>>>>>>>>>>>>>> make that clear. As it stands per the Knox > > > > >>>> documentation, > > > > >>>>>>> it > > > > >>>>>>>>>> looks > > > > >>>>>>>>>>>>> like > > > > >>>>>>>>>>>>>>>> we're going on a different route from the > > > > >>>>>>>>> preferred/recommended > > > > >>>>>>>>>>>>> idioms. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> [1] > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > http://knox.apache.org/books/knox-0-12-0/dev-guide.html#Architecture+Overview > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> Ryan, I agree about microservices. This should not > > > > >>>> derail > > > > >>>>>>> nor > > > > >>>>>>>>>> be a > > > > >>>>>>>>>>>>> major > > > > >>>>>>>>>>>>>>>> part of discussion around this feature, imho. I > > > > >>> think > > > > >>>>>>> there's > > > > >>>>>>>>>>> quite > > > > >>>>>>>>>>>> a > > > > >>>>>>>>>>>>> bit > > > > >>>>>>>>>>>>>>>> left to discuss on that subject. I want to make > > > > >> sure > > > > >>>> that > > > > >>>>>>>>> we're > > > > >>>>>>>>>>> not > > > > >>>>>>>>>>>>>>>> prematurely favoring architectural choices by > > > > >>> pulling > > > > >>>> in > > > > >>>>>>>>>> libraries > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>> are > > > > >>>>>>>>>>>>>>>> potentially opinionated about how to accomplish > > > > >>> those > > > > >>>>>>> goals. > > > > >>>>>>>>> If > > > > >>>>>>>>>>> they > > > > >>>>>>>>>>>>> are, > > > > >>>>>>>>>>>>>>>> I > > > > >>>>>>>>>>>>>>>> would expect we are comfortable ripping those > > > > >>>> libraries > > > > >>>>>> out > > > > >>>>>>>>> if > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> community favors a different direction. > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On the subject of Spring Boot vs Nodejs. I can see > > > > >>>> some > > > > >>>>>>>>>> rationale > > > > >>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>> making things homogenous (though, in a > > > > >> microservices > > > > >>>>>>>>>> architecture, > > > > >>>>>>>>>>>> if > > > > >>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>>> go > > > > >>>>>>>>>>>>>>>> that route, that's not strictly necessary), but > > > > >> what > > > > >>>> is > > > > >>>>>> the > > > > >>>>>>>>>>>>> justification > > > > >>>>>>>>>>>>>>>> for Spring Boot over Nodejs? Why would want one > > > > >> over > > > > >>>> the > > > > >>>>>>>>> other? > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> On Mon, Sep 17, 2018 at 3:38 PM Ryan Merriman < > > > > >>>>>>>>>>> merrim...@gmail.com> > > > > >>>>>>>>>>>>>>>> wrote: > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> I have reviewed a couple different PRs so I'll > > > > >> add > > > > >>>> some > > > > >>>>>>>>>> context > > > > >>>>>>>>>>>>> where I > > > > >>>>>>>>>>>>>>>>> can. Obviously Simon would be the most qualified > > > > >>> to > > > > >>>>>>> answer > > > > >>>>>>>>> but > > > > >>>>>>>>>>>> I'll > > > > >>>>>>>>>>>>>>>> add my > > > > >>>>>>>>>>>>>>>>> thoughts. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For question 1, while they may not all be > > > > >>> necessary > > > > >>>> I > > > > >>>>>>>>> think it > > > > >>>>>>>>>>>> does > > > > >>>>>>>>>>>>> make > > > > >>>>>>>>>>>>>>>>> sense to include them in this feature branch if > > > > >>> our > > > > >>>>>>> primary > > > > >>>>>>>>>> goal > > > > >>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>> integrating Knox SSO. We could push off removing > > > > >>>> JDBC > > > > >>>>>>>>>>>> authentication > > > > >>>>>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>> reasons I'll get to in my response to question > > > > >> 2. > > > > >>>> If we > > > > >>>>>>>>> want > > > > >>>>>>>>>> to > > > > >>>>>>>>>>> do > > > > >>>>>>>>>>>>> one > > > > >>>>>>>>>>>>>>>> at > > > > >>>>>>>>>>>>>>>>> a time (switch to spring boot, add Zuul as a > > > > >>>>>> dependency, > > > > >>>>>>>>> then > > > > >>>>>>>>>>> add > > > > >>>>>>>>>>>>> Knox > > > > >>>>>>>>>>>>>>>> SSO) > > > > >>>>>>>>>>>>>>>>> then that's ok but I do think there are > > > > >>> dependencies > > > > >>>>>> and > > > > >>>>>>>>>> should > > > > >>>>>>>>>>> be > > > > >>>>>>>>>>>>> done > > > > >>>>>>>>>>>>>>>> in > > > > >>>>>>>>>>>>>>>>> order. For example, adding Knox SSO requires > > > > >> some > > > > >>>> work > > > > >>>>>>>>> around > > > > >>>>>>>>>>>>> request > > > > >>>>>>>>>>>>>>>>> filtering. If we were to do this before moving > > > > >> to > > > > >>>>>> Spring > > > > >>>>>>>>> Boot > > > > >>>>>>>>>> we > > > > >>>>>>>>>>>>> would > > > > >>>>>>>>>>>>>>>>> need to implement the filters in Nodejs which > > > > >>> would > > > > >>>> be > > > > >>>>>>>>>> throwaway > > > > >>>>>>>>>>>>> once we > > > > >>>>>>>>>>>>>>>>> get around to migrating away from that. For > > > > >> Zuul, > > > > >>> I > > > > >>>>>>> believe > > > > >>>>>>>>>> it's > > > > >>>>>>>>>>>>>>>> purpose > > > > >>>>>>>>>>>>>>>>> is to facilitate the filtering (although it > > > > >> does a > > > > >>>> lot > > > > >>>>>>>>> more) > > > > >>>>>>>>>> so > > > > >>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>> doesn't > > > > >>>>>>>>>>>>>>>>> make sense to add that separate from the Knox > > > > >> SSO > > > > >>>> work. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For question 2, I think you bring up a good > > > > >> point. > > > > >>>> We > > > > >>>>>>>>> probably > > > > >>>>>>>>>>>> don't > > > > >>>>>>>>>>>>>>>> want > > > > >>>>>>>>>>>>>>>>> to just rip our current authentication method > > > > >> out. > > > > >>>> We > > > > >>>>>>> might > > > > >>>>>>>>>> want > > > > >>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>> consider deprecating it instead and making Knox > > > > >>> SSO > > > > >>>> and > > > > >>>>>>>>> LDAP > > > > >>>>>>>>>>>>>>>> authentication > > > > >>>>>>>>>>>>>>>>> optional. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For question 3, this is a bigger shift than > > > > >> just a > > > > >>>>>>>>> component > > > > >>>>>>>>>>>>> upgrade. > > > > >>>>>>>>>>>>>>>> It's > > > > >>>>>>>>>>>>>>>>> more like shifting platforms, from Elasticsearch > > > > >>> to > > > > >>>>>> Solr > > > > >>>>>>>>> for > > > > >>>>>>>>>>>>> example. > > > > >>>>>>>>>>>>>>>> Like > > > > >>>>>>>>>>>>>>>>> I alluded to in my response to question 1, I > > > > >> don't > > > > >>>>>> think > > > > >>>>>>> we > > > > >>>>>>>>>>> should > > > > >>>>>>>>>>>>>>>> require > > > > >>>>>>>>>>>>>>>>> throwaway work just because we want to review > > > > >>> these > > > > >>>>>> parts > > > > >>>>>>>>>>>>> separately. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For question 4, I will defer to Simon. I don't > > > > >>>> believe > > > > >>>>>> we > > > > >>>>>>>>>>>>> necessarily > > > > >>>>>>>>>>>>>>>>> require Zuul so I will let him elaborate on why > > > > >> he > > > > >>>>>> choose > > > > >>>>>>>>> that > > > > >>>>>>>>>>>>> library > > > > >>>>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>> what the potential impact is of adding it to our > > > > >>>>>> project. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> For question 5 and 6, I will also defer to Simon > > > > >>> on > > > > >>>>>> this. > > > > >>>>>>>>> The > > > > >>>>>>>>>>>> focus > > > > >>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>>> this feature as I understand it is a consistent > > > > >>>>>>>>> authentication > > > > >>>>>>>>>>>>> mechanism > > > > >>>>>>>>>>>>>>>>> and support for SSO. I will let him lay out his > > > > >>>> vision > > > > >>>>>>> for > > > > >>>>>>>>>> micro > > > > >>>>>>>>>>>>>>>> services. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> Knox SSO would be a great improvement and is > > > > >> what > > > > >>> I > > > > >>>>>> think > > > > >>>>>>>>> we > > > > >>>>>>>>>>>> should > > > > >>>>>>>>>>>>>>>> focus > > > > >>>>>>>>>>>>>>>>> on in this feature branch. Micro services is > > > > >>>> something > > > > >>>>>> we > > > > >>>>>>>>>> should > > > > >>>>>>>>>>>>>>>> certainly > > > > >>>>>>>>>>>>>>>>> discuss but it might be a bit of a distraction > > > > >>> and I > > > > >>>>>>>>> wouldn't > > > > >>>>>>>>>>> want > > > > >>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>> hold > > > > >>>>>>>>>>>>>>>>> up the other useful parts. > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> On Fri, Sep 14, 2018 at 8:38 PM Michael > > > > >> Miklavcic > > > > >>> < > > > > >>>>>>>>>>>>>>>>> michael.miklav...@gmail.com> wrote: > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Hey all, > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> I started looking through the Knox SSO feature > > > > >>>> branch > > > > >>>>>>>>> (see > > > > >>>>>>>>>>> here > > > > >>>>>>>>>>>>>>>>>> > > > > >>> https://issues.apache.org/jira/browse/METRON-1663 > > > > >>>> ). > > > > >>>>>>>>> This is > > > > >>>>>>>>>>>> some > > > > >>>>>>>>>>>>>>>> great > > > > >>>>>>>>>>>>>>>>> new > > > > >>>>>>>>>>>>>>>>>> security functionality work and it looks like > > > > >> it > > > > >>>> will > > > > >>>>>>>>> bring > > > > >>>>>>>>>>> some > > > > >>>>>>>>>>>>>>>>> important > > > > >>>>>>>>>>>>>>>>>> new features to the Metron platform. I'm > > > > >> coming > > > > >>> at > > > > >>>>>> this > > > > >>>>>>>>>> pretty > > > > >>>>>>>>>>>>> green, > > > > >>>>>>>>>>>>>>>> so > > > > >>>>>>>>>>>>>>>>> I > > > > >>>>>>>>>>>>>>>>>> do have some questions regarding the proposed > > > > >>>> changes > > > > >>>>>>>>> from a > > > > >>>>>>>>>>>> high > > > > >>>>>>>>>>>>>>>> level > > > > >>>>>>>>>>>>>>>>>> architectural perspective. There are a few > > > > >>> changes > > > > >>>>>>> within > > > > >>>>>>>>>> the > > > > >>>>>>>>>>>>> current > > > > >>>>>>>>>>>>>>>> FB > > > > >>>>>>>>>>>>>>>>>> PR's that I think could use some further > > > > >>>> explanation. > > > > >>>>>>> At > > > > >>>>>>>>>> first > > > > >>>>>>>>>>>>>>>> glance, it > > > > >>>>>>>>>>>>>>>>>> seems we could potentially simplify this > > > > >> branch > > > > >>> a > > > > >>>>>> great > > > > >>>>>>>>> deal > > > > >>>>>>>>>>> and > > > > >>>>>>>>>>>>> get > > > > >>>>>>>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>>>> completed much sooner if we narrowed the > > > > >> focus a > > > > >>>> bit. > > > > >>>>>>>>> But I > > > > >>>>>>>>>>>> could > > > > >>>>>>>>>>>>>>>>> certainly > > > > >>>>>>>>>>>>>>>>>> be wrong here and happy for other opinions. I > > > > >>>>>> searched > > > > >>>>>>>>>> through > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> mailing > > > > >>>>>>>>>>>>>>>>>> list history to see if there is any additional > > > > >>>>>>> background > > > > >>>>>>>>>> and > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>> main > > > > >>>>>>>>>>>>>>>>>> DISCUSS thread I could find was regarding > > > > >>>> initially > > > > >>>>>>>>> setting > > > > >>>>>>>>>> up > > > > >>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> feature > > > > >>>>>>>>>>>>>>>>>> branch, which talked about adding Knox and > > > > >> LDAP. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/cac2e6314284015b487121e77abf730abbb7ebec4ace014b19093b4c@%3Cdev.metron.apache.org%3E > > > > >>>>>>>>>>>>>>>>>> . > > > > >>>>>>>>>>>>>>>>>> If I've missed any follow-up, please let me > > > > >>> know. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Looking at the broader set of Jiras associated > > > > >>>> with > > > > >>>>>>> 1663 > > > > >>>>>>>>> and > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>> first PR > > > > >>>>>>>>>>>>>>>>>> 1665, it looks like there are 4 main thrusts > > > > >> of > > > > >>>> this > > > > >>>>>>>>> branch > > > > >>>>>>>>>>>> right > > > > >>>>>>>>>>>>> now: > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> 1. Knox/SSO > > > > >>>>>>>>>>>>>>>>>> 2. Node migrated to Spring Boot > > > > >>>>>>>>>>>>>>>>>> 3. JDBC removed completely in favor of LDAP > > > > >>>>>>>>>>>>>>>>>> 4. Introduction of Zuul, also microservices? > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> I strongly urge for the purpose of reviewing > > > > >>> this > > > > >>>>>>> feature > > > > >>>>>>>>>>> branch > > > > >>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>> we > > > > >>>>>>>>>>>>>>>>>> base much of the discussion off of > > > > >>>>>>>>>>>>>>>>>> > > > > >>> https://issues.apache.org/jira/browse/METRON-1755 > > > > >>>> , > > > > >>>>>> the > > > > >>>>>>>>>>>>> architecture > > > > >>>>>>>>>>>>>>>>>> diagram. Minimally, an explanation of the > > > > >>> current > > > > >>>>>>>>>> architecture > > > > >>>>>>>>>>>>> along > > > > >>>>>>>>>>>>>>>> with > > > > >>>>>>>>>>>>>>>>>> discussion around the additional proposed > > > > >>> changes > > > > >>>> and > > > > >>>>>>>>>>> rationale > > > > >>>>>>>>>>>>> would > > > > >>>>>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>> useful for evaluation. I don't have a solid > > > > >>> enough > > > > >>>>>>>>>>> understanding > > > > >>>>>>>>>>>>> yet > > > > >>>>>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> full scope of changes and how they differ from > > > > >>> the > > > > >>>>>>>>> existing > > > > >>>>>>>>>>>>>>>> architecture > > > > >>>>>>>>>>>>>>>>>> just from looking at the PR's alone. > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> 1. The first question is a general one > > > > >> regarding > > > > >>>> the > > > > >>>>>>>>>> necessity > > > > >>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>> 3 > > > > >>>>>>>>>>>>>>>>>> additional features alongside Knox - migrating > > > > >>>> Node > > > > >>>>>> to > > > > >>>>>>>>>> Spring > > > > >>>>>>>>>>>>> Boot, > > > > >>>>>>>>>>>>>>>>>> removing JDBC altogether, adding dependencies > > > > >> on > > > > >>>>>>>>> Netflix's > > > > >>>>>>>>>>> Zuul > > > > >>>>>>>>>>>>>>>>>> framework. > > > > >>>>>>>>>>>>>>>>>> Are these necessary for adding Knox/SSO? They > > > > >>> seem > > > > >>>>>> like > > > > >>>>>>>>>>>>> potentially > > > > >>>>>>>>>>>>>>>>>> separate features, imo. > > > > >>>>>>>>>>>>>>>>>> 2. It looks like LDAP will be a required > > > > >>> component > > > > >>>>>> for > > > > >>>>>>>>>>>> interacting > > > > >>>>>>>>>>>>>>>>> with > > > > >>>>>>>>>>>>>>>>>> Metron via the UI's. I see this PR > > > > >>>>>>>>>>>>>>>>>> https://github.com/apache/metron/pull/1186 > > > > >>> which > > > > >>>>>>> removes > > > > >>>>>>>>>> JDBC > > > > >>>>>>>>>>>>>>>>>> authentication. Are we ready to remove it > > > > >>>> completely > > > > >>>>>> or > > > > >>>>>>>>>> would > > > > >>>>>>>>>>> it > > > > >>>>>>>>>>>>> be > > > > >>>>>>>>>>>>>>>>>> better > > > > >>>>>>>>>>>>>>>>>> to leave it as a minimal installation option? > > > > >>>> What is > > > > >>>>>>> the > > > > >>>>>>>>>>>> proposed > > > > >>>>>>>>>>>>>>>>>> migration path for existing users? Do we feel > > > > >>>>>>> comfortable > > > > >>>>>>>>>>>>> requiring > > > > >>>>>>>>>>>>>>>>> that > > > > >>>>>>>>>>>>>>>>>> all installations, including full dev, install > > > > >>> and > > > > >>>>>>>>> configure > > > > >>>>>>>>>>>> LDAP? > > > > >>>>>>>>>>>>>>>> For > > > > >>>>>>>>>>>>>>>>>> comparison, in the PCAP feature branch we > > > > >>>> discussed > > > > >>>>>>>>> removing > > > > >>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> existing > > > > >>>>>>>>>>>>>>>>>> PCAP REST application in the initial > > > > >> discussion, > > > > >>>> got > > > > >>>>>>>>>>> agreement, > > > > >>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>> later > > > > >>>>>>>>>>>>>>>>>> removed it in the course of working on the > > > > >>> feature > > > > >>>>>>>>> branch. > > > > >>>>>>>>>> The > > > > >>>>>>>>>>>> PR > > > > >>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>> fairly > > > > >>>>>>>>>>>>>>>>>> clear, however I think we're just missing some > > > > >>>> basic > > > > >>>>>>>>>>> discussion > > > > >>>>>>>>>>>>>>>> around > > > > >>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> implications, as I've outlined above. Some > > > > >>>> additional > > > > >>>>>>>>>> relevant > > > > >>>>>>>>>>>>>>>>>> discussion > > > > >>>>>>>>>>>>>>>>>> occurred on this PR > > > > >>>>>>>>>>> https://github.com/apache/metron/pull/1112 > > > > >>>>>>>>>>>>>>>> which > > > > >>>>>>>>>>>>>>>>>> would be good to summarize for purposes of > > > > >> this > > > > >>>>>>>>> overarching > > > > >>>>>>>>>>>>>>>>> architecture > > > > >>>>>>>>>>>>>>>>>> discussion. > > > > >>>>>>>>>>>>>>>>>> 3. Migration from Node to Spring Boot. I > > > > >> believe > > > > >>>> this > > > > >>>>>>> is > > > > >>>>>>>>>>> already > > > > >>>>>>>>>>>>>>>> used > > > > >>>>>>>>>>>>>>>>> by > > > > >>>>>>>>>>>>>>>>>> the REST application and if anything brings > > > > >> some > > > > >>>>>>>>> cohesion to > > > > >>>>>>>>>>> our > > > > >>>>>>>>>>>>>>>>> server > > > > >>>>>>>>>>>>>>>>>> strategy. Strictly speaking, is there a reason > > > > >>>> this > > > > >>>>>> is > > > > >>>>>>>>>>> required > > > > >>>>>>>>>>>>> for > > > > >>>>>>>>>>>>>>>>>> Knox? > > > > >>>>>>>>>>>>>>>>>> It seems comparable to a component upgrade, > > > > >> such > > > > >>>> as > > > > >>>>>>>>> moving > > > > >>>>>>>>>>> from > > > > >>>>>>>>>>>> ES > > > > >>>>>>>>>>>>>>>> 2.x > > > > >>>>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> 5.6.x and upgrading Angular 6. > > > > >>>>>>>>>>>>>>>>>> 4. Introduction of Netflix's Zuul. > > > > >>>>>>>>>>>>>>>>>> > > > > >>> https://issues.apache.org/jira/browse/METRON-1665 > > > > >>>> . > > > > >>>>>>>>>>>>>>>>>> - > "The UIs currently proxy to the REST API > > > > >> to > > > > >>>> avoid > > > > >>>>>>>>> CORS > > > > >>>>>>>>>>>>>>>> issues, > > > > >>>>>>>>>>>>>>>>>> this will be achieved with Zuul." > > > > >>>>>>>>>>>>>>>>>> - Can we elaborate more on where or how CORS > > > > >> is > > > > >>> a > > > > >>>>>>> problem > > > > >>>>>>>>>> with > > > > >>>>>>>>>>>>>>>> our > > > > >>>>>>>>>>>>>>>>>> existing architecture, how Zuul will help > > > > >> solve > > > > >>>> that, > > > > >>>>>>> and > > > > >>>>>>>>>> how > > > > >>>>>>>>>>> it > > > > >>>>>>>>>>>>>>>>>> fits with > > > > >>>>>>>>>>>>>>>>>> Knox? Wouldn't this be handled by Knox? Since > > > > >>>> Larry > > > > >>>>>>> McCay > > > > >>>>>>>>>>>>>>>> chimed in > > > > >>>>>>>>>>>>>>>>>> with > > > > >>>>>>>>>>>>>>>>>> interest on the original SSO thread about the > > > > >>> FB, > > > > >>>> I'm > > > > >>>>>>>>> hoping > > > > >>>>>>>>>>> he > > > > >>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>> also > > > > >>>>>>>>>>>>>>>>>> willing to chime in on this as well. > > > > >>>>>>>>>>>>>>>>>> - This looks like it has the potential to be a > > > > >>>> rather > > > > >>>>>>>>> large > > > > >>>>>>>>>>>>>>>> piece > > > > >>>>>>>>>>>>>>>>> of > > > > >>>>>>>>>>>>>>>>>> fundamental infrastructure (as it's also > > > > >>>> pertinent to > > > > >>>>>>>>>>>>>>>>> microservices) > > > > >>>>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> pull into the platform, and I'd like to be > > > > >> sure > > > > >>>> the > > > > >>>>>>>>>> community > > > > >>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>> aware of > > > > >>>>>>>>>>>>>>>>>> and is OK with the implications. > > > > >>>>>>>>>>>>>>>>>> 5. > "The proposal is to use a spring boot > > > > >>>>>> application, > > > > >>>>>>>>>>> allowing > > > > >>>>>>>>>>>>>>>> us to > > > > >>>>>>>>>>>>>>>>>> harmonize the security implementation across > > > > >> the > > > > >>>> UI > > > > >>>>>>>>> static > > > > >>>>>>>>>>>> servers > > > > >>>>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>> the > > > > >>>>>>>>>>>>>>>>>> REST layer, and to provide a routing platform > > > > >>> for > > > > >>>>>> later > > > > >>>>>>>>>>>>>>>>> microservices." > > > > >>>>>>>>>>>>>>>>>> - > > > > >>>>>>>>>>>>>>>>>> > > > > >>> https://issues.apache.org/jira/browse/METRON-1665 > > > > >>>> . > > > > >>>>>>>>>>>>>>>>>> - Microservices is a pretty loaded term. I > > > > >> know > > > > >>>> there > > > > >>>>>>> had > > > > >>>>>>>>>> been > > > > >>>>>>>>>>>>>>>> some > > > > >>>>>>>>>>>>>>>>>> discussion a while back during the PCAP > > > > >> feature > > > > >>>>>> branch > > > > >>>>>>>>>> start, > > > > >>>>>>>>>>>>>>>> but I > > > > >>>>>>>>>>>>>>>>>> don't > > > > >>>>>>>>>>>>>>>>>> recall ever reaching a consensus on it. More > > > > >>>> detail > > > > >>>>>> in > > > > >>>>>>>>> this > > > > >>>>>>>>>>>>>>>> thread > > > > >>>>>>>>>>>>>>>>> - > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/1db7c6fa1b0f364f8c03520db9989b4f7a446de82eb4d9786055048c@%3Cdev.metron.apache.org%3E > > > > >>>>>>>>>>>>>>>>>> . > > > > >>>>>>>>>>>>>>>>>> Can we get some clarification on what is meant > > > > >>> by > > > > >>>>>>>>>>> microservices > > > > >>>>>>>>>>>>>>>>>> in the case > > > > >>>>>>>>>>>>>>>>>> of this FB and relevant PR's, what that > > > > >>>> architecture > > > > >>>>>>>>> looks > > > > >>>>>>>>>>> like, > > > > >>>>>>>>>>>>>>>>> and > > > > >>>>>>>>>>>>>>>>>> how > > > > >>>>>>>>>>>>>>>>>> it's achieved with the proposed changes in > > > > >> this > > > > >>>>>> PR/FB? > > > > >>>>>>> It > > > > >>>>>>>>>>> seems > > > > >>>>>>>>>>>>>>>>> Zuul > > > > >>>>>>>>>>>>>>>>>> is > > > > >>>>>>>>>>>>>>>>>> also pertinent to this discussion, but there > > > > >> are > > > > >>>> many > > > > >>>>>>>>> ways > > > > >>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> skin this cat > > > > >>>>>>>>>>>>>>>>>> so I don't want to presume - > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>> > > > > >>>> > > > > > https://blog.heroku.com/using_netflix_zuul_to_proxy_your_microservices > > > > >>>>>>>>>>>>>>>>>> 6. Zuul, Spring Boot, and microservices - > > > > >>> Closely > > > > >>>>>>>>> related to > > > > >>>>>>>>>>>>>>>>> point 5 > > > > >>>>>>>>>>>>>>>>>> above. It seems that we weren't quite ready > > > > >> for > > > > >>>> this > > > > >>>>>>>>> when it > > > > >>>>>>>>>>> was > > > > >>>>>>>>>>>>>>>>>> brought up > > > > >>>>>>>>>>>>>>>>>> in May, or at the very least we had some > > > > >> concern > > > > >>>> of > > > > >>>>>>> what > > > > >>>>>>>>>>>> direction > > > > >>>>>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> go. > > > > >>>>>>>>>>>>>>>>>> What is the operational impact, mpack impact, > > > > >>> and > > > > >>>> how > > > > >>>>>>> we > > > > >>>>>>>>>>> propose > > > > >>>>>>>>>>>>> to > > > > >>>>>>>>>>>>>>>>>> manage > > > > >>>>>>>>>>>>>>>>>> it with Kerberos, etc.? > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>> > > > > >>>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/c19904681e6a6d9ea3131be3d1a65b24447dca31b4aff588b263fd87@%3Cdev.metron.apache.org%3E > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> There is a lot to like in this feature branch, > > > > >>>> imo. > > > > >>>>>>> Great > > > > >>>>>>>>>>>> feature > > > > >>>>>>>>>>>>>>>>> addition > > > > >>>>>>>>>>>>>>>>>> with Knox and SSO. Introduction of LDAP > > > > >> support > > > > >>>> for > > > > >>>>>>>>>>>>> authentication for > > > > >>>>>>>>>>>>>>>>>> Metron UI's. Simplification/unification of our > > > > >>>> server > > > > >>>>>>>>>> hosting > > > > >>>>>>>>>>>>>>>>>> infrastructure. I'm hoping we can flesh out > > > > >> some > > > > >>>> of > > > > >>>>>> the > > > > >>>>>>>>>>> details > > > > >>>>>>>>>>>>>>>> pointed > > > > >>>>>>>>>>>>>>>>> out > > > > >>>>>>>>>>>>>>>>>> above a bit more and get this feature through. > > > > >>>> Great > > > > >>>>>>>>> work so > > > > >>>>>>>>>>>> far! > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>>> Best, > > > > >>>>>>>>>>>>>>>>>> Mike Miklavcic > > > > >>>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>>> -- > > > > >>>>>>>>>>>>>>> -- > > > > >>>>>>>>>>>>>>> simon elliston ball > > > > >>>>>>>>>>>>>>> @sireb > > > > >>>>>>>>>>>>>> > > > > >>>>>>>>>>>>>> -- > > > > >>>>>>>>>>>>>> -- > > > > >>>>>>>>>>>>>> simon elliston ball > > > > >>>>>>>>>>>>>> @sireb > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> ------------------- > > > > >>>>>>>>>>>>> Thank you, > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> James Sirota > > > > >>>>>>>>>>>>> PMC- Apache Metron > > > > >>>>>>>>>>>>> jsirota AT apache DOT org > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>> > > > > >>>>>>>>>> > > > > >>>>>>>>> > > > > >>>>>>>> > > > > >>>>>>> > > > > >>>>>> -- > > > > >>>>>> > > > > >>>>>> Jon Zeolla > > > > >>>> > > > > >>>> ------------------- > > > > >>>> Thank you, > > > > >>>> > > > > >>>> James Sirota > > > > >>>> PMC- Apache Metron > > > > >>>> jsirota AT apache DOT org > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > >
