Another minor improvement for users is if we pre-add $SOLR_TIP/lib to the classloader, similar to what we have with $SOLR_HOME/lib today. The disadvantage of $SOLR_HOME/lib is that it can be anywhere, perhaps on a Docker volume or a different disk, so you cannot e.g make a Dockerfile like
FROM solr:9.0 ADD foo.jar /var/solr/data/lib/foo.jar ...since /var/solr/data is a volume and will resolve to the volume partition of the user, not the content from the image. So if we instead allow users to do FROM solr:9.0 ADD foo.jar /opt/solr/lib/ That is both logical and beautiful, and would always work. I'll create a JIRA :) Jan > 13. jan. 2022 kl. 13:57 skrev Jan Høydahl <[email protected]>: > > There is not a lack of vision for future local and remote package > repositories, but the story is that package mgmt development has stalled, and > is out of reach for 1st party pkgs in the 9.0.0 timeframe. > > So we have to think progress over perfection - once again > > Phase 1. (9.0): Modularize Solr by extracting obvious low hanging fruits > plugins into contribs/modules. Make it super easy to launch solr wil any of > these on class-path (SOLR-15914 > <https://issues.apache.org/jira/browse/SOLR-15914>). > Phase 2 (9.x): Evolve package manager and make it possible to optionally > install the modules as 1st party packages instead (still fat distro) > Pase 3: (10.0?): Extract even more features as modules, and publish all > modules as separate delivery artifacts on DLCDN > > Regarding phase 2 in 9.x. We cannot really extract a feature into a module in > e.g. 9.1 so users upgrading from 9.0 will get NoClassFoundException. That > breaks back-compat. But perhaps we could continue modularization efforts in > 9.x if we make sure that all new modules extracted in a minor release are > automatically added to the classloader? Then the classes will disappear from > solr-core.jar so would possibly break someone's custom embedded usecase, but > 99% of users would be unaffected. Wdyt? > > In any case, I think for 9.x the realistic route is to keep our fat tgz, but > make it slimmer by removing redundancy and prune down on the number of > overlapping dependencies. That can get us a long way. > > Jan > >> 13. jan. 2022 kl. 03:15 skrev David Smiley <[email protected] >> <mailto:[email protected]>>: >> >> Shawn: >> * RE redundancies of stuff in /dist/, see >> https://issues.apache.org/jira/browse/SOLR-15916 >> <https://issues.apache.org/jira/browse/SOLR-15916> >> * RE "contrib" vs "module" vs "package", see: >> https://issues.apache.org/jira/browse/SOLR-15917 >> <https://issues.apache.org/jira/browse/SOLR-15917> >> * RE not shipping these extras with the Solr distribution, see: "slim >> distro" mention in the document "Solr first party packages" >> https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing >> >> <https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing> >> >> It could very well be worth shipping two docker images in the meantime. >> Or maybe a zip of each module could be a separate artifact that is >> published? I'm not sure what freedoms we have to do this in the ASF. >> >> ~ David Smiley >> Apache Lucene/Solr Search Developer >> http://www.linkedin.com/in/davidwsmiley >> <http://www.linkedin.com/in/davidwsmiley> >> >> On Wed, Jan 12, 2022 at 8:21 PM Shawn Heisey <[email protected] >> <mailto:[email protected]>> wrote: >> On 1/12/2022 8:31 AM, Jan Høydahl wrote: >> > I think there are lots of pieces of code in solr-core that can easily be >> > extracted the same way. >> > Some perhaps even for 9.0.0, as it slims down the core and reduces attack >> > surface for most users as well. >> >> I think it would be really awesome if we had a core download that only >> included basic functionality, and all the other fancy things that Solr >> does now out of the box (as well as those that are contrib) could be >> added after download via package scripting or just additional downloads. >> >> The size of solr-8.11.1.tgz is 207MiB, or 218076598 bytes. The .zip >> version is slightly larger. 8.0.0 was 163MiB, 7.0.0 was 142MiBm, 6.0.0 >> was 131MiB, and 1.4.1 was 53.7MiB. I think it's insane that the >> download is so big ... and a lot of what makes it big are things that >> the vast majority of our users will never use. >> >> Large reductions in the overall size of the main download would be >> possible by putting hadoop, calcite, some of the really large lucene >> analysis components, and the contrib stuff into packages. The >> extraction contrib alone is 43.5MiB compressed in zip format. >> >> I would suggest moving zookeeper and its dependencies as well, but I >> think we probably want SolrCloud to be part of base functionality. >> >> Some of the large jars are included for what are probably insignificant >> usages, and I wonder if that functionality could be replaced by newer >> native functions available in Java 8 and later. I am eyeballing things >> like guava and the commons-* jars here, but I am sure there are other >> things in this category. I'd like to eliminate as many dependencies as >> we can. >> >> Extracting some things from the solr-core jar into other jars sounds >> like a really awesome idea. >> >> I don't think the solr-core jar should be in the dist directory. It's >> useless by itself, because it will still have a LOT of dependencies even >> if we shrink it. And there are likely other things in the dist >> directory that fall into that category. The test framework and its >> dependencies are a good candidate for removal. >> >> By removing some of the low-hanging fruit that I am SURE isn't needed >> for base binary functionality on the 8.11.1 download, I was able to end >> up with a .zip file sized in at 60.4MiB, and I am sure at least a little >> bit of further reduction is possible if we can fully map out >> dependencies. I think we can leverage gradle to provide some dependency >> info. >> >> Exactly how to organize the code repo to create divided artifacts is >> something that we would need to think about. My initial idea is >> changing "contrib" to "package" and then making some new directories >> under package. >> >> Thanks, >> Shawn >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >
