Yes, please go through SIP before inviting us to review the actual PR. And once you re-work the SIP page based on feedback and converge towards consensus, I’d recommend starting a formal VOTE thread for the SIP, as outlined in the SIP page. That way people can have a chance to speak up on the overall design so that does not become a topic once again in the PR.
My wish for the SIP is to be very clear on exactly how the UI is proposed shipped and whether any manual steps are needed to enable/install. Also try to clarify whether you propose a big-bang replace or whether the old and new UI will need to co-exists. Jan > 14. apr. 2020 kl. 15:29 skrev Cassandra Targett <[email protected]>: > > Marcus, > > A couple thoughts… > > First, sorry you’re having problems with Confluence. I suspect the issue is > permissions. There are only two groups allowed to add pages to the SOLR > space, “lucene” and “lucene-pmc”. I believe these correspond to ASF LDAP > groups, which would mean they include committers and PMC members only. We can > grant you individual permission to add/edit pages, however; we’ve done this > for a handful of others. I could do this for you, just ping me off-thread so > I can confirm your username. > > I don’t know what happened to the pages you tried to create. You can try to > see if they are hidden somehow, maybe in your profile’s “Recently worked on” > section (https://cwiki.apache.org/confluence/dashboard.action#recently-worked > <https://cwiki.apache.org/confluence/dashboard.action#recently-worked>). When > you’re on that page, the menu on the left also shows a “Saved for later” page > that maybe has them. > > At any rate, I wouldn’t suggest putting all the design decisions/discussion > into a PR. That’s what we traditionally used Jira for. We decided to use SIPs > to make navigating design discussions in Jira easier - decisions would get > lost in comments - and to forestall someone doing possibly wasted work until > they have some degree of community agreement for what they hope to do. If you > put it in a PR, by its very nature those decisions would already be made and > that could lead to significant amounts of rework. > > The SIP process is to write up the SIP and then file a Jira issue, and you > can’t file a PR without a Jira, so you’ll need a Jira issue for this work > anyway. If you really can’t write a SIP, then the fallback is Jira, not a PR. > > Re: the API issues - those should also go into Jira. I can’t see your list so > far since it requires permission, but I’ll just say I would not recommend > dropping your whole doc into a single Jira. They’ll need to be broken out > into separate ones (and it’s pretty likely issues already exist for at least > some of the things). I know that sounds like a PITA, but we don’t track > issues in personal Google docs, we use Jira. And if some of the items are > possibly controversial, then individual items for each one will allow us to > work through the controversies and not stall progress on things that are not > problematic (if anyone is so inclined to work on those). > > Hopefully you don’t mind the unsolicited advice on this, just trying to help > you understand some of our ways of doing things. > > Cassandra > On Apr 14, 2020, 3:18 AM -0500, Marcus Eagan <[email protected]>, wrote: >> Mike and Gus, >> >> I tried to share my SIP after writing it, and then it disappeared. I also >> tried to write some and save it, but then it disappeared. I also tried to >> write it outside the wiki and paste it again. >> >> I'm not sure what's going on, but I will try again tomorrow I suppose. If I >> fail again, my SIP will probably come in the form of a Pull Request, with >> videos, screenshots, a lit of todos, and helpful documentation so that >> people can get started easily. >> >> From this project, I have also uncovered what I would consider some serious >> issues with the Solr API and some other issues that I dig through Jira for >> before I bother the community. I will do my best to document them all and >> open tickets. Some times I rage quit, walk away, come back and forget. I >> will try to fix some of them, but some of them that I have looked into look >> like worm cans. It might be good if a few people here are prepared to >> explore fixing the API in a few areas, but I'm not sure about the best way >> to go about doing that with angering lots of people. Seeking advice on how >> to approach the API challenges. I don't just want to start complaining about >> things, nor do I want to take on everything. A small group from a variety of >> different organizations to discuss some of these challenges might be most >> helpful. On the flip side, I suspect things will improve in other areas as a >> result. >> >> You can view progress here: >> https://drive.google.com/file/d/1NgO34DRp1llMp3EwJQcKBn2LM4r5PD39/view?usp=sharing >> >> <https://drive.google.com/file/d/1NgO34DRp1llMp3EwJQcKBn2LM4r5PD39/view?usp=sharing> >> >> I've mostly managed to get all the requests sorted, and rendering of the >> data (mostly). i didn't get much time today as I was very busy at work. And >> tomorrow I get my quarantine checkup and will probably be sleepy early. But >> by the end of next weekend, I should have finished everything but the query >> editor and all the other associated collections screens which I will bucket >> under collections. That current dropdown menu is less intuitive than optimal >> but I understand. >> >> Soliciting feedback now because once I open the PR I am hoping we have >> agreed on as many things as possible. If no one suggest query view designs, >> I will mock some up and share them. Again, I repeat, not a designer. But I >> did take couple classes, and have built products from start to finish mostly >> on an island so I will be able to manage. Looking for feedback that the >> community might be getting from its clients. Obviously, I cannot accommodate >> all the requests, but if something is recurring, I want to support the users >> to support the adoption of this new UI. >> >> As if I needed to say it, and many of you probably suspected this was >> coming: Erick this is a lot of work. Holy shit. And not just because I'm a >> product manager. It would be challenging for anyone. This project, like all >> project, has some skeletons. From my perspective, that's gfreat because >> there are lots of things to improve. Also, the project is still pretty >> amazing. >> >> Thanks again for your help everyone, >> >> Marcus >> >> >> On Mon, Apr 13, 2020 at 9:55 AM Marcus Eagan <[email protected] >> <mailto:[email protected]>> wrote: >> Gus, >> >> SIP sounds good. I will share. >> >> Marcus >> >> >> >> On Mon, Apr 13, 2020 at 09:13 Gus Heck <[email protected] >> <mailto:[email protected]>> wrote: >> Maybe start collecting Design and Design choices in a SIP? This discussion >> has been good and there seems to be consensus that we want a new UI, we want >> it to be a package and we want the package to be available by default and >> well tested. "Package" seems to imply that it can be added or removed or >> replaced or an alternative UI installed along side of it. If we got all of >> those things done this would be amazingly awesome :) >> >> Another thing that would be valuable is a good doc that explains "how to >> edit and maintain the UI", written for an audience that is experienced in SW >> dev but not UI development (probably including some basics around framework >> chosen). This could be in a README or in the "dev docs" that has been >> mentioned elsewhere. >> >> The SIP would be a great place to elaborate on technology choices & supply a >> link to things like the video :) >> >> On Mon, Apr 13, 2020 at 10:35 AM Mike Drob <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Marcus, >> >> The mailing list strips attachments for some folks, can you upload the video >> somewhere else and link to it for us poor unfortunate souls? Thanks for your >> work! Excited to see the progress as it happens. >> >> Mike >> >> On Mon, Apr 13, 2020 at 5:30 AM Marcus Eagan <[email protected] >> <mailto:[email protected]>> wrote: >> In general, I asked for some degree of trouble when I volunteered for this >> work. Don't beat me too hard. My primary goal is to achieve three things: >> >> 1) Improve security when using Solr Admin UI by removing EOL, unsupported >> code. >> 2) Make it easier or more welcoming for new developers to try the project >> and even become contributors in all areas of the project because the UI >> looks and functions as slightly more contemporary. >> 3) Give back more substantially to a community from which I have received so >> much with a testable and perrty UI. >> >> I've added another (4) which is contribute to help make the package manager >> a first-class citizen in the minds of many Solr users around the world via >> the UI package. I will need some help from someone in this list on deploying >> this UI with a jar in Gradle if we want to offer an alternative option to >> install the UI in Solr 9. I've attached where I'm at. It's a nights and >> weekends project, but I will always be available for bug fixes or >> discussions, unless i'm in a meeting or reading a book 😇 >> >> I won't solicit a ton of feedback prior to the the first PR, which I will >> leave open for a few weeks or even a couple months while I put some lipstick >> on it and improve performance of the application >> >> <<<<<<< Over the weekend read lots of documentation, wrote a bunch of code >> when it wasn't holidays, built the services, and stumbled through the logic >> of rendering all these data points so you can watch the attached video if >> you want to check it out. >>>>>>>> >> >> . There's definitely some areas where I didn't do the TypeScript thing >> because I'm still trying to grok it a bit.The two areas of the that I am >> looking to somewhat overhaul in potentially controversial ways are the >> queries page and actual flow of the collections experience, which at the >> moment are sort of linked, yet disconnected at the same time. The >> Collections page tab today is really an Alias page. It won't have its own >> tab in the new application is the plan, unless someone can give me a good >> reason. Almost everything else will stay the same. >> >> For that reason, I'd like to solicit feedback if anyone has any examples or >> ideas they'd like to share, I would greatly appreciate it. I'm somewhat far >> along with the Admin UI as of now. Short weekend because of holiday >> activities and general quarantine craziness, but I'm maybe 25-35% of the way >> to completion, depending on how much care I devote to the query screen. >> >> Even though there are some big problems with Angular — some major ones — I >> think this was the right way to go for many reasons. I'm about a quarter as >> fast in Angular as I am in React, but this is the right decision for the >> long haul. I can elaborate if anyone really cares. Most importantly, this >> app will be a lot easier to maintain. >> >> But now I have a handle on TypeScript. Many optimizations and improvements >> coming to Solr as a result. TypeScript can be pretty opinionated about APIs >> for someone thinking that they are back in the wild wild west. >> >> Sending big air hugs, >> Marcus >> >> On Fri, Apr 10, 2020 at 16:20 Noble Paul <[email protected] >> <mailto:[email protected]>> wrote: >> +1 to David Smiley >> >> On Sat, Apr 11, 2020, 8:12 AM Marcus Eagan <[email protected] >> <mailto:[email protected]>> wrote: >> I don't see admin UI as non-core. I think that an application UI for >> end-users of an application consumes Solr non-core. I have to resign from >> arguing, though. >> >> I don't consider myself a UI expert. I can do the work. >> >> >> On Fri, Apr 10, 2020 at 11:42 AM Ishan Chattopadhyaya >> <[email protected] <mailto:[email protected]>> wrote: >> David, you capture my thoughts well. >> >> Having a UI as a package gives users more choice and gives our users more >> flexibility. >> 1. Users would be able to use a latter version of the UI with an older >> version of Solr, or vice versa. >> 2. Users should be able to install multiple types of UI, from different >> publishers, at once. >> 3. Contributors should be able to contribute to the UI more easily, since >> collaboration can be less bureaucratic. Experts like Marcus won't need to >> depend on preoccupied committers like us. >> 4. A UI (not the default one) can use libraries that aren't even Apache 2.0 >> compatible. >> 5. We can setup and use UI test frameworks for test automation (selenium >> etc), that would be challenging to setup and maintain with ASF Jenkins. >> >> List goes on.. >> >> Whether the package is a first party or third party can be a separate >> discussion. There should be an extremely easy and well defined way (support >> in the script itself) to start Solr with the packaged UI enabled. >> >> In any case, I don't think it is conducive to let UI code be part of the >> Solr's core codebase, where it currently is. The reason is, we can't fix >> bugs if we break something. We don't have automated testing either to know >> whether or not we broke anything. >> >> Every healthy project has a rich plugin ecosystem, and such non core >> improvements should be delivered via packages. >> >> On Fri, 10 Apr, 2020, 8:33 pm Marcus Eagan, <[email protected] >> <mailto:[email protected]>> wrote: >> I agree with you. 😃 >> >> On Fri, Apr 10, 2020 at 06:22 David Smiley <[email protected] >> <mailto:[email protected]>> wrote: >> I disagree that "package management frameworks are for loading non-essential >> features or features not enabled by default". >> >> I don't think the proposal of the UI being a "package" (in the new package >> system) implies that the UI (or _any_ package) is not a highly valuable >> package that is so highly valuable that we want it installed by default. >> Noble and I were brainstorming on some ideas where even much of Solr's >> internal instances of plugin interfaces (e.g. query parsers, etc.) might >> even be a new "core" package or some-such. The value in putting much of >> Solr in a package, 1st party, is separation of concerns. and better >> classpath management. >> >> I think it's "essential" that a UI ship with Solr by default -- meaning, >> without the user having to take any additional steps whatsoever. As Jan >> said it's been this way a long time. >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> <http://www.linkedin.com/in/davidwsmiley> >> >> On Fri, Apr 10, 2020 at 1:35 AM Marcus Eagan <[email protected] >> <mailto:[email protected]>> wrote: >> I think package management frameworks are for loading non-essential features >> or features not enabled by default. On essentialness, experts should not >> decide what is essential based on how they use a system. They should >> consider the community of users. Regarding UI, it is and should be enabled >> by default. Only a few use cases prefer it to be disabled and some of those >> are because of its current state. They would like to use it in an updated >> form. >> >> What is the technical rationale that outweighs the needs and behaviors of >> our users to strip the user interface out of Solr? >> >> Thank you Noble and everyone else, >> Marcus >> >> >> On Thu, Apr 9, 2020 at 19:06 Noble Paul <[email protected] >> <mailto:[email protected]>> wrote: >> My 2 cents (again) >> >> if packages are disabled by default , how will UI work? >> >> We can make an exception for this one and enable only this by default >> >> Do we test the UI and certify it? >> >> The UI package can be shipped along with Solr distro ,like the million >> other jars that we ship with Solr today and every version of Solr can >> be certified for a certain version of the UI package. We should have >> sanity tests to ensure that the given version of UI works well with >> Solr. "My commit has broken the UI and it's not my problem" should not >> be a valid excuse. The UI sanity tests should pass as a part of the >> tests. >> >> Is the UI important? >> >> Yes, the admin UI is the face of Solr for may users. People always >> assumed it existed & they depend on it. >> >> The current admin UI has fallen behind. If the new UI effort delivers >> on the promise, this is a great opportunity to get rid of that old >> baggage & make Solr codebase even slimmer >> >> On Fri, Apr 10, 2020 at 6:06 AM Jan Høydahl <[email protected] >> <mailto:[email protected]>> wrote: >> > >> > Solr has always had an admin UI and if anyone wants to propose it should >> > not, please start another thread or vote about that, and do not divert >> > this thread which is about how to improve and future proof the Admin UI. >> > >> > I believe the Admin UI should be strengthened and enhanced, not removed. >> > It can perfectly well be an official and even default on part of every >> > release, perfectly in sync. Whether it is in core as today or a package or >> > a stand-alone process or a new webapp, are then really what we discuss >> > here. >> > >> > Perhaps after people have voiced their opinions in this thread, a SIP can >> > be crafted with a concrete plan. We can then have a vote on the SIP. >> > >> > Jan Høydahl >> > >> > > 9. apr. 2020 kl. 20:06 skrev Cassandra Targett <[email protected] >> > > <mailto:[email protected]>>: >> > > >> > > >> > > Thanks for your message, Gus. You touched on things I was thinking this >> > > morning as I caught up to the thread, and had started to draft a message >> > > about. >> > > >> > > I feel like there is an assumption underlying some of our discussion >> > > about packages that says a feature or whatever has to either part of our >> > > core codebase or 100% maintained by someone “outside” the community (by >> > > which I mean someone who is not a committer and/or operates completely >> > > independently from the rest of the project activities). >> > > >> > > But it’s important to remember there’s nothing inherent in the package >> > > concept that says a package can't be wholly >> > > maintained/distributed/supported by the Lucene PMC and our community of >> > > committers and contributors. It’s not uncommon for software to have a >> > > base package of the core software and also plugins that most users would >> > > consider “essential”, maintained by the same people making the core, but >> > > which are added after the base install. There would be details to work >> > > out in terms of how we manage that, but none of those are technically >> > > impossible. I don’t think we only have a binary choice (3rd party >> > > package or part of Solr core). >> > > >> > > In fact, I’d go so far as to say I suspect the only way packages are >> > > going to see any real traction is if we take the lead in creating and >> > > maintaining a few to show the way forward for everyone else who may be >> > > interested in doing the same. The package concept introduces an idea of >> > > a Solr ecosystem that has not really existed to date and like all new >> > > ecosystems (communities), it needs some degree of nurturing to grow or >> > > it will not take off. >> > > >> > > To bring it back around to the UI, though, I agree we need to decide: is >> > > a UI important? I would argue that it is. I regularly talk to users >> > > whose Solr knowledge and experience are quite advanced yet who still >> > > rely entirely on the UI to carry out basic tasks. It’s just easier - >> > > less to remember, don’t need to look up a command, etc. Persistent >> > > problems with performance of the UI in large clusters and gaping holes >> > > in functionality are deeply frustrating to those users. >> > > >> > > Because it is important to our users, even if any new UI ends up living >> > > outside our community, it will need our explicit approval in some form >> > > or we’re going to hear complaints until the end of time that Solr >> > > doesn’t have a UI (or worse, we “took away” the UI). Without our ongoing >> > > endorsement and blessing it will just be another toy maintained by >> > > “somebody” (no offense, Marcus) until he is forced to abandon it due to >> > > lack of user interest and/or lack of personal time to do all the heavy >> > > lifting by himself. >> > > >> > > Cassandra >> > >> On Apr 9, 2020, 11:32 AM -0500, Erick Erickson <[email protected] >> > >> <mailto:[email protected]>>, wrote: >> > >> Gus: >> > >> >> > >> Very thoughtful post. I think you raise an _extremely_ important point >> > >> about “how critical is the UI?” And by extension other packages. If >> > >> they’re critical to Solr, the question of how to keep them in sync >> > >> becomes paramount. >> > >> >> > >> I agree that the admin UI is important, if we have a mechanism to >> > >> insure it’s kept in sync with the release that would be near the to of >> > >> my list. >> > >> >> > >> Best, >> > >> Erick >> > >> >> > >>> On Apr 9, 2020, at 12:06 PM, Gus Heck <[email protected] >> > >>> <mailto:[email protected]>> wrote: >> > >>> >> > >>> In my view this brings us up against a bit of an existential question. >> > >>> How do we ensure quality of packages that are key to Solr? I'm sure >> > >>> that there are folks who don't find the UI very useful, but it's >> > >>> important to others. The rationale that "elastic keeps their's >> > >>> separate" has to be tempered by the actual real differences between >> > >>> Elastic and Solr. Elastic has a corporate sponsor, a coordinated road >> > >>> map, and explicitly ensures that all the bits that are maintained >> > >>> separately work together (so that they don't have excessive support >> > >>> costs or bad first experiences that impact their bottom line etc.). >> > >>> >> > >>> Solr is in a different place however, and we need to carefully examine >> > >>> the question of whether something that works for Elastic works for us. >> > >>> Lucidworks and several other large companies do spend a lot of money >> > >>> on developers that contribute to Solr, but there is no organization >> > >>> around multiple components that MUST work together. Another example of >> > >>> this is Solr and Lucene, and our defense against a lack of >> > >>> coordination of components in that case has been to unify them into a >> > >>> single release package. >> > >>> >> > >>> So IMHO we need to answer 2 questions: >> > >>> 1.) Do we consider the UI important. If I'm alone or in a minority in >> > >>> feeling it's important, then so be it and it doesn't really matter >> > >>> what we do. (maybe a vote?) >> > >>> 2.) If we make it "a package" how do we ensure that important packages >> > >>> such as the UI are never broken by a new release. >> > >>> >> > >>> IMHO I don't thing we should tolerate a situation where things we >> > >>> consider important are broken frequently. >> > >>> >> > >>> For my part obviously my answer to 1 is "yes" :). As for 2, one thing >> > >>> that comes to mind is what the Ant project did (may still do?) with >> > >>> (and my memory from 15 years ago is foggy here, so forgive me if >> > >>> something I say is not quite perfect) the GUMP build server that ran >> > >>> ant builds for a bunch of different projects that depended on ant to >> > >>> ensure early detections of changes that would break existing projects. >> > >>> If we have a good UI test suite and commit to that being part of the >> > >>> release build package that might be a solution. I honestly don't >> > >>> actually care where it lives, but I do think it hurts us if it becomes >> > >>> broken and unusable, or hard to install. >> > >>> >> > >>> My worry is that "Solr developers are not UI developers" is really >> > >>> code-words for "I want to be able to break it and let others clean up >> > >>> the mess, because I'm not a UI developer". I have this worry with >> > >>> respect to all "packages", but I may be missing info from discussions >> > >>> about the package system, which initiated during a very busy period >> > >>> for me so background links that I should have read are welcome if I've >> > >>> got something wrong here :) I went looking for a SIP but didn't see >> > >>> it... I have found a google doc linked from SOLR-13661 >> > >>> >> > >>> Finally to a detail about one of the above suggestions the option to >> > >>> automatically download and install the UI could be good, but that then >> > >>> requires that packages be available from somewhere that never goes >> > >>> away like maven central, or that Apache commits to hosting a >> > >>> repository server indefinitely, but again that's surely been discussed >> > >>> WRT packages already... Using Github in such a way is subject to being >> > >>> broken arbitrarily when Github decides to restrict things for cost >> > >>> reasons (ask Bower about that one WRT rate limiting...) or the >> > >>> "repository" has to be something local and therefore must be included >> > >>> part of the distribution... at which point it's still a thing we >> > >>> distribute and since we're distributing it and we probably don't mean >> > >>> to distribute broken stuff we still need UI developers... >> > >>> >> > >>> Also, I thought the package loading stuff was supposed to be disabled >> > >>> by default for security, that seems to conflict with or at least >> > >>> complicate the notion of easily installing as a package. >> > >>> >> > >>> So "package" is a good for modularizing code, or for 3rd party >> > >>> (possibly paid) plugins (I have a client that might find that >> > >>> interesting in the future) but we have to ensure that it doesn't lead >> > >>> to a lack of maintenance for things that are critical. >> > >>> >> > >>> Incidentally though I've said I favor Angular CLI, (significantly >> > >>> because I've got some start on learning it) it also occurs to me that >> > >>> perhaps anything "modern" is a difficulty because those things all >> > >>> have a learning curve, and maximizing accessibility and ease of >> > >>> modifications for folks not steeped in UI development might be our >> > >>> priority (different from the priorities a commercial site would have). >> > >>> The flip side argument is that with a popular framework, it would be >> > >>> easier for UI focused folks to contribute... but will they? and does >> > >>> that leave us perennially rewriting the UI in whatever is popular? >> > >>> (maybe that's ok?) I think in all our decisions here we need to be >> > >>> very careful to distinguish how our needs may differ in unusual ways >> > >>> from the needs of commercial web development. >> > >>> >> > >>> -Gus >> > >>> >> > >>> On Thu, Apr 9, 2020 at 8:14 AM Erick Erickson <[email protected] >> > >>> <mailto:[email protected]>> wrote: >> > >>> Marcus: >> > >>> >> > >>> re-reading the thread, it looks to me like the consensus from Noble >> > >>> and Ishan and Jan is that as long as the new, nifty UI is a separate >> > >>> package, go ahead and knock yourself out ;). The objection is to >> > >>> making it part of the Solr code base… We’ll all be thrilled with if we >> > >>> can rip the current admin UI out ;) >> > >>> >> > >>> That said, I suspect it’ll be one of the tighter packages. It’d be >> > >>> super-cool if we could run the UI tests on Jenkins say once a day just >> > >>> to keep it up to date. >> > >>> >> > >>> The admin UI has always been somewhat awkwardly bolted on the side of >> > >>> Solr, it’d be great to have it have a more elegant architecture. >> > >>> >> > >>> The other exciting thing would be that clients could then use the >> > >>> package code as something they can incorporate/fork/whatever. >> > >>> Practically every client I’ve worked with at large installations has >> > >>> rolled their own dashboard. If they could use a package as a starting >> > >>> point, it’d be welcome. >> > >>> >> > >>> Best, >> > >>> Erick >> > >>> >> > >>>> On Apr 9, 2020, at 3:07 AM, Marcus Eagan <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> >> > >>>> Hey Noble, >> > >>>> >> > >>>> -1 is a definitive, so I want to clarify that you are saying you do >> > >>>> not wish to remove the EOL front end and replace it with another one >> > >>>> in the longer term? >> > >>>> >> > >>>> I hear you! As a product manager in my day job, my primary goal is to >> > >>>> find features to cut! I spend a lot of time thinking about >> > >>>> non-essential vs used heavily vs causes more problems than it's >> > >>>> worth. I can tell you from watching the many people in the field at >> > >>>> Lucidworks, there are a lot of people who know quite a bit about >> > >>>> Solr, but rely on the Admin UI heavily because they feel comfortable >> > >>>> there. Those people in effect help us stay employed despite never >> > >>>> contributing or being capable of contributing to Solr. So hear me >> > >>>> out. I've got a proposal: >> > >>>> >> > >>>> To start, I can work on this app as an optional package for your >> > >>>> awesome new package manager. It will be the second one I've worked on >> > >>>> in my evenings and weekends btw. The first was a package validator >> > >>>> that I hope to eventually open source, but its complexity and lack of >> > >>>> popularity because it is security ;( will likely make it the second >> > >>>> one I open source/finish. I'm also collaborating with a couple >> > >>>> members of the Lucidworks security team on that one, but I have built >> > >>>> the basics already for them to build upon. >> > >>>> >> > >>>> Back to the new UI discussion and my update that I promised. >> > >>>> >> > >>>> My update was going to be that after evaluating the projects Jan >> > >>>> posted, the most recent project that Jan listed created a pretty good >> > >>>> base to build on. After lots of auditing of the packages and a bit of >> > >>>> refactoring because the UI world moves fast, I was able to get it to >> > >>>> transpile and run again (as I'm sure it previously did) and from >> > >>>> (2290 vulns): >> > >>>> <image.png> >> > >>>> no npm fix doesn't magically fix, I wish it did >> > >>>> >> > >>>> to (2 low sev, non-productions vulns): >> > >>>> <image.png> >> > >>>> >> > >>>> These two issues will not affect production and really only the unit >> > >>>> tests. Besides, I plan to remove them before we get to a stage even >> > >>>> that matters. I've also started investigating the level of effort for >> > >>>> me to get it to feature parity with the current app. The preliminary >> > >>>> answer is not very much compared to other work I've done in shorter >> > >>>> time — working with a jerk for a boss (years ago, don't worry >> > >>>> Hatcher). I'm building a couple of the missing features as we speak. >> > >>>> From the beginning, it will have infinitely more test coverage and >> > >>>> will be a lot more approachable to contemporary UI developers. It >> > >>>> will also make the Solr experience for new developers simpler. The >> > >>>> major design changes that I have been thinking about would be to the >> > >>>> cloud view and the query view. Both of those are important, the first >> > >>>> to more experienced users, and the second to less advanced users >> > >>>> though occasionally an advanced debugger or demo presenter in my >> > >>>> experience. >> > >>>> >> > >>>> In the end Noble, this is about making Solr more approachable to new >> > >>>> users not experts like you. The growth of Solr adoption only benefits >> > >>>> you, so I would ask you to revisit your -1 at some point in the near >> > >>>> future when you see the progress and breadth of improvement. We have >> > >>>> had customers complain about the Admin UI and the community has even >> > >>>> complained about it. I think this is the right thing to do. If you >> > >>>> still consider effectively upgrading the current Admin UI as feature >> > >>>> creep, I can revisit the package manager compromise or move the >> > >>>> efforts elsewhere. I respect your position. A search service is >> > >>>> nothing without a strong and diverse set of skills and capabilities >> > >>>> behind it and making it accessible to everyone who needs it. >> > >>>> >> > >>>> For those who care, here's my 4 Node cluster of tech products running >> > >>>> locally. >> > >>>> >> > >>>> <image.png> >> > >>>> >> > >>>> The shoulders of the homie that put that scaffold together are broad! >> > >>>> Props to him. I will volunteer to put together extensive docs for JS >> > >>>> devs who want to contribute and make it better once we get it to a >> > >>>> place where it replaces and improves upon the current option. I'll >> > >>>> even sponsor some prizes for college kids or people recently out of >> > >>>> work to get cranking on this bad boy. >> > >>>> >> > >>>> Thank you Noble and everyone else, >> > >>>> >> > >>>> Marcus >> > >>>> >> > >>>> On Wed, Apr 8, 2020 at 11:20 PM Noble Paul <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> Solr developers are not UI experts. We are a search service and such >> > >>>> a service should have nice clean APIs + documentation. Is a UI useful >> > >>>> ? yes. The last thing we want today is another complex component in >> > >>>> Solr codebase that nobody understands or cannot maintain. >> > >>>> >> > >>>> So, Solr UI can be hosted outside our codebase and we can have an >> > >>>> option to install UI from that remote repo >> > >>>> >> > >>>> something like "bin/solr install-gui" >> > >>>> >> > >>>> I'm -1 on anymore feature creep. >> > >>>> >> > >>>> --Noble >> > >>>> >> > >>>> >> > >>>> On Thu, Apr 9, 2020 at 12:22 PM Marcus Eagan <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> Thanks again Gus. >> > >>>> >> > >>>> Lots of people indeed misuse REST so we could go on and on about >> > >>>> whether requests are stateless or not in another thread. Let's spare >> > >>>> the group. >> > >>>> >> > >>>> I think most everyone on this channel would be in agreement with you >> > >>>> on separate app. I'll be opening a new ticket and a PR that will >> > >>>> document a few things to make it easy for UI devs who know little to >> > >>>> no Java how to get started. >> > >>>> >> > >>>> Ishan, there's some significant UI expertise in the team. Erickson >> > >>>> finds his way to open every cookie jar. Erik Hatcher wrote the first >> > >>>> version of Blacklight. I've seen Pugh do lots of work on Quepid's UI. >> > >>>> Jan and Kevin have done a lot of work, and so have many others. The >> > >>>> list goes on, and *likes to work on UI* is a different discussion. >> > >>>> >> > >>>> Beyond committers because I'm not a committer, I have UI expertise >> > >>>> that I can polish off and improve for the sake of my interest and >> > >>>> commitment to the community and I like to do it. I've also led UI >> > >>>> teams. I can help to steward the effort overall and keep things up to >> > >>>> date up to the point where I need to ask one of the committers to >> > >>>> help me get changes merged. I'll probably even hire a developer to >> > >>>> work on it once we are to that point. ;-) >> > >>>> >> > >>>> Expertise is not something that should block us but motivate us to >> > >>>> expand this community and/or our own skillsets long term. >> > >>>> >> > >>>> Thank you both and everyone else, >> > >>>> >> > >>>> Marcus >> > >>>> >> > >>>> On Wed, Apr 8, 2020, 10:21 AM Gus Heck <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> While running it in an external node does ensure separability, I >> > >>>> don't think it does a good job of addressing my other point of not >> > >>>> needing to manage a 3rd server. It's still a server if it's started >> > >>>> by java, and one still has to ensure it exists, and it will be extra >> > >>>> hard to figure out how to configure it if started by Solr. >> > >>>> >> > >>>> I'm strongly in favor of us having a UI from my perspective as a >> > >>>> consultant it makes discovery of things like their startup parameters >> > >>>> and directories and such very easy (just go to front page of the >> > >>>> admin screen), and it's so much easier to get a customer with >> > >>>> security concerns and strict controls on who can access what (think >> > >>>> banks, military, etc) to share a web session where they drive the UI >> > >>>> than to get direct access to machines. It'll be a lot slower and much >> > >>>> lower service to be making people wait while I craft curl statements >> > >>>> to paste into the web session (and then fix the inevitable typos, or >> > >>>> detect when they missed the last char of what I pasted, etc...). >> > >>>> >> > >>>> I definitely against Solr spawning some other server (node or >> > >>>> otherwise) on it's own and thereby requiring additional system >> > >>>> dependencies, or creating a second process that needs to be >> > >>>> configured and properly secured. To me that's even worse than >> > >>>> requiring the UI to run outside of Solr. We have a perfectly good web >> > >>>> container already, and furthermore there's a much greater likelihood >> > >>>> that maintainers will be facile with java/j2ee than anything else >> > >>>> (IMHO). It's great if the framework we choose uses little or no >> > >>>> JSP/Servlet and is modernized with a 100% javascript, templated etc. >> > >>>> front end, but the back end should be java/jetty because we've got >> > >>>> lots of java folks. >> > >>>> >> > >>>> If the back end matters deeply then you're not really programming to >> > >>>> MVC/REST style... >> > >>>> >> > >>>> So there's another $0.02 :) and if you're not careful I'll give you >> > >>>> an entire nickle's worth of ways people misuse/misunderstand the term >> > >>>> REST :) >> > >>>> >> > >>>> -Gus >> > >>>> >> > >>>> On Tue, Apr 7, 2020 at 9:06 PM Marcus Eagan <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> Gus, >> > >>>> >> > >>>> Your $.02 are worth a lot more than $.02 USD, so thank you. >> > >>>> >> > >>>> By separate app, I think I mean to endorse managed by a Node.js >> > >>>> process started by NPM. I don’t think that conflicts with what you >> > >>>> have proposed. The NPM command should be issued by Java || or Bash >> > >>>> but I don’t think it would add significant overhead. Also, seems like >> > >>>> on CI and or precommit hooks front end could be sizzled in parallel >> > >>>> without adding much overhead. >> > >>>> >> > >>>> As for the front end framework, the most important things to consider >> > >>>> in my view are simplicity and maintainability. We need to do a >> > >>>> thorough analysis on the ecosystem and issues like the size of a >> > >>>> React project vs Angular project vs Vue project, but React and Vue >> > >>>> certainly have the velocity and the hearts if the front end community >> > >>>> more than Angular. React is MIT license now and for the foreseeable >> > >>>> future thanks to the power and reach of its developers. >> > >>>> >> > >>>> <[email protected] <mailto:[email protected]>> wrote: >> > >>>> +1 for Angular CLI / Typescript since I've fiddled with this in a >> > >>>> minor way recently, Also MIT license is super friendly. >> > >>>> >> > >>>> >> > >>>> As a disenfranchised volunteer to the project, I also assume voters >> > >>>> on specific choices like frameworks will be helping build in some >> > >>>> respect at some point now or in the future. Is that a fair or >> > >>>> misguided assumption? >> > >>>> >> > >>>> Marcus >> > >>>> >> > >>>> On Tue, Apr 7, 2020 at 17:15 Gus Heck <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> +1 for Angular CLI / Typescript since I've fiddled with this in a >> > >>>> minor way recently, Also MIT license is super friendly. >> > >>>> >> > >>>> Separate App - hmm... that's got some attraction, but also gives my >> > >>>> stomach some churning when I think about solr now requiring >> > >>>> management of 3 different servers (solr, something to serve UI and >> > >>>> zookeeper). Adding more infrastructure gives me pause with respect to >> > >>>> all the smaller installations. I've had several small self funded >> > >>>> startup clients and a few clients with existing initial installs that >> > >>>> they are outgrowing in places where procuring new machines and new >> > >>>> software is a 6-12 mo endeavor and both types seem to squirm when I >> > >>>> make suggestions such as running zookeeper separately, (let alone 3 >> > >>>> of them). I think separate looks good for medium to large folks or >> > >>>> very large companies that **already have** a solr expert on hand, but >> > >>>> hurts the small clients and the departments in large orgs that got >> > >>>> started with insufficient advice/expertise, so maybe >> > >>>> >> > >>>> - The UI should be installed by default >> > >>>> - it should be easy to remove it, or start with it disabled >> > >>>> - it should be self contained and separately downloadable. >> > >>>> >> > >>>> My recent fiddling included figuring out how to make angular CLI play >> > >>>> nice in a J2ee war file structure seen here: >> > >>>> https://github.com/nsoft/ns-login <https://github.com/nsoft/ns-login> >> > >>>> >> > >>>> By play nice I mean, >> > >>>> - build creates a war file that "just works" when installed >> > >>>> - Angluar CLI commands work >> > >>>> - Angular serve command works (for auto-reloading ui changes, running >> > >>>> on port 4200; note the use of proxy to allow it to talk to an already >> > >>>> running web container) >> > >>>> >> > >>>> My $0.02, >> > >>>> >> > >>>> -Gus >> > >>>> >> > >>>> On Mon, Apr 6, 2020 at 11:03 AM Jörn Franke <[email protected] >> > >>>> <mailto:[email protected]>> wrote: >> > >>>> I think standalone would be very useful. >> > >>>> I propose Angular with Typescript - it fits to a more data centric >> > >>>> approach with data types etc. >> > >>>> Maybe even two types of UIs - Admin UI and a simple Search UI. >> > >>>> >> > >>>> >> > >>>>> Am 06.04.2020 um 16:53 schrieb Jan Høydahl <[email protected] >> > >>>>> <mailto:[email protected]>>: >> > >>>>> >> > >>>>> Thanks for kickstarting this and bringing some fresh blood and >> > >>>>> enthusiasm :) >> > >>>>> >> > >>>>> Looks like others have had similar wish for a standalone Solr Admin >> > >>>>> App, here’s a quick GitHub search for inspiration: >> > >>>>> >> > >>>>> https://github.com/savantly-net/solr-admin >> > >>>>> <https://github.com/savantly-net/solr-admin> (Angular, nice >> > >>>>> screenshots, 1y old) >> > >>>>> https://github.com/kezhenxu94/yasa >> > >>>>> <https://github.com/kezhenxu94/yasa> (vuejs, impressive screenshots, >> > >>>>> 2y old) >> > >>>>> https://github.com/thereactleague/galaxy >> > >>>>> <https://github.com/thereactleague/galaxy> (React, no screenshots, >> > >>>>> 4y old) >> > >>>>> >> > >>>>> They all seem abandoned but perhaps a new official effort could >> > >>>>> bring their developers in as contributors again? >> > >>>>> >> > >>>>>> the people who work on the Admin UI do not need to be expected to >> > >>>>>> know the Java workflow, necessarily. This reality widens the net >> > >>>>>> for who can contribute. >> > >>>>> >> > >>>>> Agree. Frontend devs have been a shortage in this project, and if we >> > >>>>> can make it easier to attract UI committers who feel at home and >> > >>>>> productive with the UI code, that would be a win. On the other hand, >> > >>>>> if we expect that the UI will be maintained by regular Java >> > >>>>> committers, then anything that makes it easier for them/us to >> > >>>>> contribute is also a win, like perhaps strongly-typed. >> > >>>>> >> > >>>>> Again, thanks Marcus for reviving this topic. Let us all try not to >> > >>>>> be overly ambitious here or shoot the initiative down with >> > >>>>> bikeshedding. It is far more important to fuel the energy and >> > >>>>> momentum and get something built than to remain stuck :) >> > >>>>> >> > >>>>> Jan >> > >>>>> >> > >>>>> >> > >>>>>> 6. apr. 2020 kl. 13:47 skrev Marcus Eagan <[email protected] >> > >>>>>> <mailto:[email protected]>>: >> > >>>>>> >> > >>>>>> Coming back to these existential questions from my phone: >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> Jan Høydahl >> > >>>>>> Added 1 hour ago >> > >>>>>> There are many opinions around admin UI. So I think the best place >> > >>>>>> to start would be a new mail-thread in dev@ to discuss the way >> > >>>>>> forward. Before we start a major re-work, we should probably ask >> > >>>>>> ourselves a few existential questions: >> > >>>>>> • Should we turn Amin UI into a standalone app instead of embedded >> > >>>>>> in Solr? >> > >>>>>> >> > >>>>>> I think it should be a standalone app. There are many advantages >> > >>>>>> gained from a separation of such concerns. Some of the ones >> > >>>>>> include, the people who work on the Admin UI do not need to be >> > >>>>>> expected to know the Java workflow, necessarily. This reality >> > >>>>>> widens the net for who can contribute. >> > >>>>>> >> > >>>>>> Testing becomes a lot easier because JS developers are accustomed >> > >>>>>> to building tests for static assets and self-contained node apps. >> > >>>>>> They generally know less about testing a bit of JS within a massive >> > >>>>>> Java project. The test could also run independently for changes >> > >>>>>> that only affect the front end. Adding test coverage without adding >> > >>>>>> time to tests sounds awesome. >> > >>>>>> >> > >>>>>> There are quite a few tickets over the years that have seemed to >> > >>>>>> suggest that people want more fine-grained control over the Solr >> > >>>>>> admin UI overall. Two recent tickets discussed topics like running >> > >>>>>> a Solr Admin app on only one node and disabling it al together for >> > >>>>>> whatever reason. See: >> > >>>>>> https://issues.apache.org/jira/browse/SOLR-14014 >> > >>>>>> <https://issues.apache.org/jira/browse/SOLR-14014>. >> > >>>>>> >> > >>>>>> • What UI framework? Guess anything is better than current EOL, but >> > >>>>>> will largely depend on who is willing to do the job! >> > >>>>>> I’m happy to take this on (and willing to follow through on >> > >>>>>> completing in my nights and weekends), but I am mostly framework >> > >>>>>> agnostic. My stronge preference would be React, provided the >> > >>>>>> license is kosher. There was one blip of “practically unusable for >> > >>>>>> most orgs” a couple years back, but Facebook made it right really >> > >>>>>> soon after. However, I’m flexible. Angular (not JS) and Vue are >> > >>>>>> also great. I would recommend we consider Typescript also because >> > >>>>>> of the size of project and number of strongly-typed devs on this >> > >>>>>> mailing list. My only reservation with TypeScript, though it may >> > >>>>>> not apply in this case, is that the supersets of JS have changed a >> > >>>>>> lot more than the frameworks. While CoffeeScript was an unnecessary >> > >>>>>> layer of abstraction from my limited perspective, TypeScript might >> > >>>>>> make JS more embraceable to a list of Java hackers. >> > >>>>>> >> > >>>>>> • Current UI has no test coverage, can we do better with the new UI? >> > >>>>>> >> > >>>>>> It’s imperative.React, Angular, and Vue each make it easy to >> > >>>>>> include tests. >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> https://issues.apache.org/jira/browse/SOLR-12276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076204#comment-17076204 >> > >>>>>> >> > >>>>>> <https://issues.apache.org/jira/browse/SOLR-12276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17076204#comment-17076204> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> >> > >>>> (work) >> > >>>> http://www.the111shift.com <http://www.the111shift.com/> (play) >> > >>>> -- >> > >>>> Marcus Eagan >> > >>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> >> > >>>> (work) >> > >>>> http://www.the111shift.com <http://www.the111shift.com/> (play) >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> ----------------------------------------------------- >> > >>>> Noble Paul >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> Marcus Eagan >> > >>>> >> > >>> >> > >>> >> > >>> --------------------------------------------------------------------- >> > >>> To unsubscribe, e-mail: [email protected] >> > >>> <mailto:[email protected]> >> > >>> For additional commands, e-mail: [email protected] >> > >>> <mailto:[email protected]> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work) >> > >>> http://www.the111shift.com <http://www.the111shift.com/> (play) >> > >> >> > >> >> > >> --------------------------------------------------------------------- >> > >> To unsubscribe, e-mail: [email protected] >> > >> <mailto:[email protected]> >> > >> For additional commands, e-mail: [email protected] >> > >> <mailto:[email protected]> >> > >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > <mailto:[email protected]> >> > For additional commands, e-mail: [email protected] >> > <mailto:[email protected]> >> > >> >> >> -- >> ----------------------------------------------------- >> Noble Paul >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >> -- >> Marcus Eagan >> >> -- >> Marcus Eagan >> >> >> >> -- >> Marcus Eagan >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >> -- >> http://www.needhamsoftware.com <http://www.needhamsoftware.com/> (work) >> http://www.the111shift.com <http://www.the111shift.com/> (play) >> -- >> Marcus Eagan >> >> >> >> -- >> Marcus Eagan >>
