Thanks for your message, Gus. You touched on things I was thinking this morning as I caught up to the thread, and had started to draft a message about.
I feel like there is an assumption underlying some of our discussion about packages that says a feature or whatever has to either part of our core codebase or 100% maintained by someone “outside” the community (by which I mean someone who is not a committer and/or operates completely independently from the rest of the project activities). But it’s important to remember there’s nothing inherent in the package concept that says a package can't be wholly maintained/distributed/supported by the Lucene PMC and our community of committers and contributors. It’s not uncommon for software to have a base package of the core software and also plugins that most users would consider “essential”, maintained by the same people making the core, but which are added after the base install. There would be details to work out in terms of how we manage that, but none of those are technically impossible. I don’t think we only have a binary choice (3rd party package or part of Solr core). In fact, I’d go so far as to say I suspect the only way packages are going to see any real traction is if we take the lead in creating and maintaining a few to show the way forward for everyone else who may be interested in doing the same. The package concept introduces an idea of a Solr ecosystem that has not really existed to date and like all new ecosystems (communities), it needs some degree of nurturing to grow or it will not take off. To bring it back around to the UI, though, I agree we need to decide: is a UI important? I would argue that it is. I regularly talk to users whose Solr knowledge and experience are quite advanced yet who still rely entirely on the UI to carry out basic tasks. It’s just easier - less to remember, don’t need to look up a command, etc. Persistent problems with performance of the UI in large clusters and gaping holes in functionality are deeply frustrating to those users. Because it is important to our users, even if any new UI ends up living outside our community, it will need our explicit approval in some form or we’re going to hear complaints until the end of time that Solr doesn’t have a UI (or worse, we “took away” the UI). Without our ongoing endorsement and blessing it will just be another toy maintained by “somebody” (no offense, Marcus) until he is forced to abandon it due to lack of user interest and/or lack of personal time to do all the heavy lifting by himself. Cassandra On Apr 9, 2020, 11:32 AM -0500, Erick Erickson <erickerick...@gmail.com>, wrote: > Gus: > > Very thoughtful post. I think you raise an _extremely_ important point about > “how critical is the UI?” And by extension other packages. If they’re > critical to Solr, the question of how to keep them in sync becomes paramount. > > I agree that the admin UI is important, if we have a mechanism to insure it’s > kept in sync with the release that would be near the to of my list. > > Best, > Erick > > > On Apr 9, 2020, at 12:06 PM, Gus Heck <gus.h...@gmail.com> wrote: > > > > In my view this brings us up against a bit of an existential question. How > > do we ensure quality of packages that are key to Solr? I'm sure that there > > are folks who don't find the UI very useful, but it's important to others. > > The rationale that "elastic keeps their's separate" has to be tempered by > > the actual real differences between Elastic and Solr. Elastic has a > > corporate sponsor, a coordinated road map, and explicitly ensures that all > > the bits that are maintained separately work together (so that they don't > > have excessive support costs or bad first experiences that impact their > > bottom line etc.). > > > > Solr is in a different place however, and we need to carefully examine the > > question of whether something that works for Elastic works for us. > > Lucidworks and several other large companies do spend a lot of money on > > developers that contribute to Solr, but there is no organization around > > multiple components that MUST work together. Another example of this is > > Solr and Lucene, and our defense against a lack of coordination of > > components in that case has been to unify them into a single release > > package. > > > > So IMHO we need to answer 2 questions: > > 1.) Do we consider the UI important. If I'm alone or in a minority in > > feeling it's important, then so be it and it doesn't really matter what we > > do. (maybe a vote?) > > 2.) If we make it "a package" how do we ensure that important packages such > > as the UI are never broken by a new release. > > > > IMHO I don't thing we should tolerate a situation where things we consider > > important are broken frequently. > > > > For my part obviously my answer to 1 is "yes" :). As for 2, one thing that > > comes to mind is what the Ant project did (may still do?) with (and my > > memory from 15 years ago is foggy here, so forgive me if something I say is > > not quite perfect) the GUMP build server that ran ant builds for a bunch of > > different projects that depended on ant to ensure early detections of > > changes that would break existing projects. If we have a good UI test suite > > and commit to that being part of the release build package that might be a > > solution. I honestly don't actually care where it lives, but I do think it > > hurts us if it becomes broken and unusable, or hard to install. > > > > My worry is that "Solr developers are not UI developers" is really > > code-words for "I want to be able to break it and let others clean up the > > mess, because I'm not a UI developer". I have this worry with respect to > > all "packages", but I may be missing info from discussions about the > > package system, which initiated during a very busy period for me so > > background links that I should have read are welcome if I've got something > > wrong here :) I went looking for a SIP but didn't see it... I have found a > > google doc linked from SOLR-13661 > > > > Finally to a detail about one of the above suggestions the option to > > automatically download and install the UI could be good, but that then > > requires that packages be available from somewhere that never goes away > > like maven central, or that Apache commits to hosting a repository server > > indefinitely, but again that's surely been discussed WRT packages > > already... Using Github in such a way is subject to being broken > > arbitrarily when Github decides to restrict things for cost reasons (ask > > Bower about that one WRT rate limiting...) or the "repository" has to be > > something local and therefore must be included part of the distribution... > > at which point it's still a thing we distribute and since we're > > distributing it and we probably don't mean to distribute broken stuff we > > still need UI developers... > > > > Also, I thought the package loading stuff was supposed to be disabled by > > default for security, that seems to conflict with or at least complicate > > the notion of easily installing as a package. > > > > So "package" is a good for modularizing code, or for 3rd party (possibly > > paid) plugins (I have a client that might find that interesting in the > > future) but we have to ensure that it doesn't lead to a lack of maintenance > > for things that are critical. > > > > Incidentally though I've said I favor Angular CLI, (significantly because > > I've got some start on learning it) it also occurs to me that perhaps > > anything "modern" is a difficulty because those things all have a learning > > curve, and maximizing accessibility and ease of modifications for folks not > > steeped in UI development might be our priority (different from the > > priorities a commercial site would have). The flip side argument is that > > with a popular framework, it would be easier for UI focused folks to > > contribute... but will they? and does that leave us perennially rewriting > > the UI in whatever is popular? (maybe that's ok?) I think in all our > > decisions here we need to be very careful to distinguish how our needs may > > differ in unusual ways from the needs of commercial web development. > > > > -Gus > > > > On Thu, Apr 9, 2020 at 8:14 AM Erick Erickson <erickerick...@gmail.com> > > wrote: > > Marcus: > > > > re-reading the thread, it looks to me like the consensus from Noble and > > Ishan and Jan is that as long as the new, nifty UI is a separate package, > > go ahead and knock yourself out ;). The objection is to making it part of > > the Solr code base… We’ll all be thrilled with if we can rip the current > > admin UI out ;) > > > > That said, I suspect it’ll be one of the tighter packages. It’d be > > super-cool if we could run the UI tests on Jenkins say once a day just to > > keep it up to date. > > > > The admin UI has always been somewhat awkwardly bolted on the side of Solr, > > it’d be great to have it have a more elegant architecture. > > > > The other exciting thing would be that clients could then use the package > > code as something they can incorporate/fork/whatever. Practically every > > client I’ve worked with at large installations has rolled their own > > dashboard. If they could use a package as a starting point, it’d be welcome. > > > > Best, > > Erick > > > > > On Apr 9, 2020, at 3:07 AM, Marcus Eagan <marcusea...@gmail.com> wrote: > > > > > > Hey Noble, > > > > > > -1 is a definitive, so I want to clarify that you are saying you do not > > > wish to remove the EOL front end and replace it with another one in the > > > longer term? > > > > > > I hear you! As a product manager in my day job, my primary goal is to > > > find features to cut! I spend a lot of time thinking about non-essential > > > vs used heavily vs causes more problems than it's worth. I can tell you > > > from watching the many people in the field at Lucidworks, there are a lot > > > of people who know quite a bit about Solr, but rely on the Admin UI > > > heavily because they feel comfortable there. Those people in effect help > > > us stay employed despite never contributing or being capable of > > > contributing to Solr. So hear me out. I've got a proposal: > > > > > > To start, I can work on this app as an optional package for your awesome > > > new package manager. It will be the second one I've worked on in my > > > evenings and weekends btw. The first was a package validator that I hope > > > to eventually open source, but its complexity and lack of popularity > > > because it is security ;( will likely make it the second one I open > > > source/finish. I'm also collaborating with a couple members of the > > > Lucidworks security team on that one, but I have built the basics already > > > for them to build upon. > > > > > > Back to the new UI discussion and my update that I promised. > > > > > > My update was going to be that after evaluating the projects Jan posted, > > > the most recent project that Jan listed created a pretty good base to > > > build on. After lots of auditing of the packages and a bit of refactoring > > > because the UI world moves fast, I was able to get it to transpile and > > > run again (as I'm sure it previously did) and from (2290 vulns): > > > <image.png> > > > no npm fix doesn't magically fix, I wish it did > > > > > > to (2 low sev, non-productions vulns): > > > <image.png> > > > > > > These two issues will not affect production and really only the unit > > > tests. Besides, I plan to remove them before we get to a stage even that > > > matters. I've also started investigating the level of effort for me to > > > get it to feature parity with the current app. The preliminary answer is > > > not very much compared to other work I've done in shorter time — working > > > with a jerk for a boss (years ago, don't worry Hatcher). I'm building a > > > couple of the missing features as we speak. From the beginning, it will > > > have infinitely more test coverage and will be a lot more approachable to > > > contemporary UI developers. It will also make the Solr experience for new > > > developers simpler. The major design changes that I have been thinking > > > about would be to the cloud view and the query view. Both of those are > > > important, the first to more experienced users, and the second to less > > > advanced users though occasionally an advanced debugger or demo presenter > > > in my experience. > > > > > > In the end Noble, this is about making Solr more approachable to new > > > users not experts like you. The growth of Solr adoption only benefits > > > you, so I would ask you to revisit your -1 at some point in the near > > > future when you see the progress and breadth of improvement. We have had > > > customers complain about the Admin UI and the community has even > > > complained about it. I think this is the right thing to do. If you still > > > consider effectively upgrading the current Admin UI as feature creep, I > > > can revisit the package manager compromise or move the efforts elsewhere. > > > I respect your position. A search service is nothing without a strong and > > > diverse set of skills and capabilities behind it and making it accessible > > > to everyone who needs it. > > > > > > For those who care, here's my 4 Node cluster of tech products running > > > locally. > > > > > > <image.png> > > > > > > The shoulders of the homie that put that scaffold together are broad! > > > Props to him. I will volunteer to put together extensive docs for JS devs > > > who want to contribute and make it better once we get it to a place where > > > it replaces and improves upon the current option. I'll even sponsor some > > > prizes for college kids or people recently out of work to get cranking on > > > this bad boy. > > > > > > Thank you Noble and everyone else, > > > > > > Marcus > > > > > > On Wed, Apr 8, 2020 at 11:20 PM Noble Paul <noble.p...@gmail.com> wrote: > > > Solr developers are not UI experts. We are a search service and such a > > > service should have nice clean APIs + documentation. Is a UI useful ? > > > yes. The last thing we want today is another complex component in Solr > > > codebase that nobody understands or cannot maintain. > > > > > > So, Solr UI can be hosted outside our codebase and we can have an option > > > to install UI from that remote repo > > > > > > something like "bin/solr install-gui" > > > > > > I'm -1 on anymore feature creep. > > > > > > --Noble > > > > > > > > > On Thu, Apr 9, 2020 at 12:22 PM Marcus Eagan <marcusea...@gmail.com> > > > wrote: > > > Thanks again Gus. > > > > > > Lots of people indeed misuse REST so we could go on and on about whether > > > requests are stateless or not in another thread. Let's spare the group. > > > > > > I think most everyone on this channel would be in agreement with you on > > > separate app. I'll be opening a new ticket and a PR that will document a > > > few things to make it easy for UI devs who know little to no Java how to > > > get started. > > > > > > Ishan, there's some significant UI expertise in the team. Erickson finds > > > his way to open every cookie jar. Erik Hatcher wrote the first version of > > > Blacklight. I've seen Pugh do lots of work on Quepid's UI. Jan and Kevin > > > have done a lot of work, and so have many others. The list goes on, and > > > *likes to work on UI* is a different discussion. > > > > > > Beyond committers because I'm not a committer, I have UI expertise that I > > > can polish off and improve for the sake of my interest and commitment to > > > the community and I like to do it. I've also led UI teams. I can help to > > > steward the effort overall and keep things up to date up to the point > > > where I need to ask one of the committers to help me get changes merged. > > > I'll probably even hire a developer to work on it once we are to that > > > point. ;-) > > > > > > Expertise is not something that should block us but motivate us to expand > > > this community and/or our own skillsets long term. > > > > > > Thank you both and everyone else, > > > > > > Marcus > > > > > > On Wed, Apr 8, 2020, 10:21 AM Gus Heck <gus.h...@gmail.com> wrote: > > > While running it in an external node does ensure separability, I don't > > > think it does a good job of addressing my other point of not needing to > > > manage a 3rd server. It's still a server if it's started by java, and one > > > still has to ensure it exists, and it will be extra hard to figure out > > > how to configure it if started by Solr. > > > > > > I'm strongly in favor of us having a UI from my perspective as a > > > consultant it makes discovery of things like their startup parameters and > > > directories and such very easy (just go to front page of the admin > > > screen), and it's so much easier to get a customer with security concerns > > > and strict controls on who can access what (think banks, military, etc) > > > to share a web session where they drive the UI than to get direct access > > > to machines. It'll be a lot slower and much lower service to be making > > > people wait while I craft curl statements to paste into the web session > > > (and then fix the inevitable typos, or detect when they missed the last > > > char of what I pasted, etc...). > > > > > > I definitely against Solr spawning some other server (node or otherwise) > > > on it's own and thereby requiring additional system dependencies, or > > > creating a second process that needs to be configured and properly > > > secured. To me that's even worse than requiring the UI to run outside of > > > Solr. We have a perfectly good web container already, and furthermore > > > there's a much greater likelihood that maintainers will be facile with > > > java/j2ee than anything else (IMHO). It's great if the framework we > > > choose uses little or no JSP/Servlet and is modernized with a 100% > > > javascript, templated etc. front end, but the back end should be > > > java/jetty because we've got lots of java folks. > > > > > > If the back end matters deeply then you're not really programming to > > > MVC/REST style... > > > > > > So there's another $0.02 :) and if you're not careful I'll give you an > > > entire nickle's worth of ways people misuse/misunderstand the term REST :) > > > > > > -Gus > > > > > > On Tue, Apr 7, 2020 at 9:06 PM Marcus Eagan <marcusea...@gmail.com> wrote: > > > Gus, > > > > > > Your $.02 are worth a lot more than $.02 USD, so thank you. > > > > > > By separate app, I think I mean to endorse managed by a Node.js process > > > started by NPM. I don’t think that conflicts with what you have proposed. > > > The NPM command should be issued by Java || or Bash but I don’t think it > > > would add significant overhead. Also, seems like on CI and or precommit > > > hooks front end could be sizzled in parallel without adding much overhead. > > > > > > As for the front end framework, the most important things to consider in > > > my view are simplicity and maintainability. We need to do a thorough > > > analysis on the ecosystem and issues like the size of a React project vs > > > Angular project vs Vue project, but React and Vue certainly have the > > > velocity and the hearts if the front end community more than Angular. > > > React is MIT license now and for the foreseeable > > > future thanks to the power and reach of its developers. > > > > > > <gus.h...@gmail.com> wrote: > > > +1 for Angular CLI / Typescript since I've fiddled with this in a minor > > > way recently, Also MIT license is super friendly. > > > > > > > > > As a disenfranchised volunteer to the project, I also assume voters on > > > specific choices like frameworks will be helping build in some respect at > > > some point now or in the future. Is that a fair or misguided assumption? > > > > > > Marcus > > > > > > On Tue, Apr 7, 2020 at 17:15 Gus Heck <gus.h...@gmail.com> wrote: > > > +1 for Angular CLI / Typescript since I've fiddled with this in a minor > > > way recently, Also MIT license is super friendly. > > > > > > Separate App - hmm... that's got some attraction, but also gives my > > > stomach some churning when I think about solr now requiring management of > > > 3 different servers (solr, something to serve UI and zookeeper). Adding > > > more infrastructure gives me pause with respect to all the smaller > > > installations. I've had several small self funded startup clients and a > > > few clients with existing initial installs that they are outgrowing in > > > places where procuring new machines and new software is a 6-12 mo > > > endeavor and both types seem to squirm when I make suggestions such as > > > running zookeeper separately, (let alone 3 of them). I think separate > > > looks good for medium to large folks or very large companies that > > > **already have** a solr expert on hand, but hurts the small clients and > > > the departments in large orgs that got started with insufficient > > > advice/expertise, so maybe > > > > > > - The UI should be installed by default > > > - it should be easy to remove it, or start with it disabled > > > - it should be self contained and separately downloadable. > > > > > > My recent fiddling included figuring out how to make angular CLI play > > > nice in a J2ee war file structure seen here: > > > https://github.com/nsoft/ns-login > > > > > > By play nice I mean, > > > - build creates a war file that "just works" when installed > > > - Angluar CLI commands work > > > - Angular serve command works (for auto-reloading ui changes, running on > > > port 4200; note the use of proxy to allow it to talk to an already > > > running web container) > > > > > > My $0.02, > > > > > > -Gus > > > > > > On Mon, Apr 6, 2020 at 11:03 AM Jörn Franke <jornfra...@gmail.com> wrote: > > > I think standalone would be very useful. > > > I propose Angular with Typescript - it fits to a more data centric > > > approach with data types etc. > > > Maybe even two types of UIs - Admin UI and a simple Search UI. > > > > > > > > > > Am 06.04.2020 um 16:53 schrieb Jan Høydahl <jan....@cominvent.com>: > > > > > > > > Thanks for kickstarting this and bringing some fresh blood and > > > > enthusiasm :) > > > > > > > > Looks like others have had similar wish for a standalone Solr Admin > > > > App, here’s a quick GitHub search for inspiration: > > > > > > > > https://github.com/savantly-net/solr-admin (Angular, nice screenshots, > > > > 1y old) > > > > https://github.com/kezhenxu94/yasa (vuejs, impressive screenshots, 2y > > > > old) > > > > https://github.com/thereactleague/galaxy (React, no screenshots, 4y old) > > > > > > > > They all seem abandoned but perhaps a new official effort could bring > > > > their developers in as contributors again? > > > > > > > > > the people who work on the Admin UI do not need to be expected to > > > > > know the Java workflow, necessarily. This reality widens the net for > > > > > who can contribute. > > > > > > > > Agree. Frontend devs have been a shortage in this project, and if we > > > > can make it easier to attract UI committers who feel at home and > > > > productive with the UI code, that would be a win. On the other hand, if > > > > we expect that the UI will be maintained by regular Java committers, > > > > then anything that makes it easier for them/us to contribute is also a > > > > win, like perhaps strongly-typed. > > > > > > > > Again, thanks Marcus for reviving this topic. Let us all try not to be > > > > overly ambitious here or shoot the initiative down with bikeshedding. > > > > It is far more important to fuel the energy and momentum and get > > > > something built than to remain stuck :) > > > > > > > > Jan > > > > > > > > > > > > > 6. apr. 2020 kl. 13:47 skrev Marcus Eagan <m...@marcuseagan.com>: > > > > > > > > > > Coming back to these existential questions from my phone: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jan Høydahl > > > > > Added 1 hour ago > > > > > There are many opinions around admin UI. So I think the best place to > > > > > start would be a new mail-thread in dev@ to discuss the way forward. > > > > > Before we start a major re-work, we should probably ask ourselves a > > > > > few existential questions: > > > > > • Should we turn Amin UI into a standalone app instead of embedded in > > > > > Solr? > > > > > > > > > > I think it should be a standalone app. There are many advantages > > > > > gained from a separation of such concerns. Some of the ones include, > > > > > the people who work on the Admin UI do not need to be expected to > > > > > know the Java workflow, necessarily. This reality widens the net for > > > > > who can contribute. > > > > > > > > > > Testing becomes a lot easier because JS developers are accustomed to > > > > > building tests for static assets and self-contained node apps. They > > > > > generally know less about testing a bit of JS within a massive Java > > > > > project. The test could also run independently for changes that only > > > > > affect the front end. Adding test coverage without adding time to > > > > > tests sounds awesome. > > > > > > > > > > There are quite a few tickets over the years that have seemed to > > > > > suggest that people want more fine-grained control over the Solr > > > > > admin UI overall. Two recent tickets discussed topics like running a > > > > > Solr Admin app on only one node and disabling it al together for > > > > > whatever reason. See: > > > > > https://issues.apache.org/jira/browse/SOLR-14014. > > > > > > > > > > • What UI framework? Guess anything is better than current EOL, but > > > > > will largely depend on who is willing to do the job! > > > > > I’m happy to take this on (and willing to follow through on > > > > > completing in my nights and weekends), but I am mostly framework > > > > > agnostic. My stronge preference would be React, provided the license > > > > > is kosher. There was one blip of “practically unusable for most orgs” > > > > > a couple years back, but Facebook made it right really soon after. > > > > > However, I’m flexible. Angular (not JS) and Vue are also great. I > > > > > would recommend we consider Typescript also because of the size of > > > > > project and number of strongly-typed devs on this mailing list. My > > > > > only reservation with TypeScript, though it may not apply in this > > > > > case, is that the supersets of JS have changed a lot more than the > > > > > frameworks. While CoffeeScript was an unnecessary layer of > > > > > abstraction from my limited perspective, TypeScript might make JS > > > > > more embraceable to a list of Java hackers. > > > > > > > > > > • Current UI has no test coverage, can we do better with the new UI? > > > > > > > > > > It’s imperative.React, Angular, and Vue each make it easy to include > > > > > tests. > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/browse/SOLR-12276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076204#comment-17076204 > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > http://www.needhamsoftware.com (work) > > > http://www.the111shift.com (play) > > > -- > > > Marcus Eagan > > > > > > > > > > > > -- > > > http://www.needhamsoftware.com (work) > > > http://www.the111shift.com (play) > > > > > > > > > -- > > > ----------------------------------------------------- > > > Noble Paul > > > > > > > > > -- > > > Marcus Eagan > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: dev-h...@lucene.apache.org > > > > > > > > -- > > http://www.needhamsoftware.com (work) > > http://www.the111shift.com (play) > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org >
