> > It could very well be worth shipping two docker images in the meantime. > Or maybe a zip of each module could be a separate artifact that is > published? I'm not sure what freedoms we have to do this in the ASF. >
I think for 9.0 we could realistically shoot for 2 binary releases and 2 docker images, slim (without the modules) and full-featured (with the modules), having the full-featured be the default. Starting in the 9.x line, we could start packaging the modules as separate binary artifacts for the solr release. Then in 10.x we can make the slim release be the default (still having the fat tgz available as well with as solr-extended-10.0.0.tgz or something like that). > Phase 1. (9.0): Modularize Solr by extracting obvious low hanging fruits > plugins into contribs/modules. Make it super easy to launch solr wil any of > these on class-path (SOLR-15914 > <https://issues.apache.org/jira/browse/SOLR-15914>). > Phase 2 (9.x): Evolve package manager and make it possible to optionally > install the modules as 1st party packages instead (still fat distro) > Pase 3: (10.0?): Extract even more features as modules, and publish all > modules as separate delivery artifacts on DLCDN > I really like this plan. I agree for 9.x we really don't have an option, but to keep publishing the fat tgz as the default. Even in 10.x I think we want to offer both a full-featured download and a slim download, but with first-part-packages we can make slim the "default". Another minor improvement for users is if we pre-add $SOLR_TIP/lib to the > classloader I'll create a JIRA :) Yes please. That would be a lovely improvement! People bend-over-backward currently to add custom libs. - Houston On Thu, Jan 13, 2022 at 8:09 AM Jan Høydahl <jan....@cominvent.com> wrote: > Another minor improvement for users is if we pre-add $SOLR_TIP/lib to the > classloader, similar to what we have with $SOLR_HOME/lib today. The > disadvantage of $SOLR_HOME/lib is that it can be anywhere, perhaps on a > Docker volume or a different disk, so you cannot e.g make a Dockerfile like > > FROM solr:9.0 > ADD foo.jar /var/solr/data/lib/foo.jar > > ...since /var/solr/data is a volume and will resolve to the volume > partition of the user, not the content from the image. So if we instead > allow users to do > > FROM solr:9.0 > ADD foo.jar /opt/solr/lib/ > > That is both logical and beautiful, and would always work. > > I'll create a JIRA :) > > Jan > > 13. jan. 2022 kl. 13:57 skrev Jan Høydahl <jan....@cominvent.com>: > > There is not a lack of vision for future local and remote package > repositories, but the story is that package mgmt development has stalled, > and is out of reach for 1st party pkgs in the 9.0.0 timeframe. > So we have to think progress over perfection - once again > > Phase 1. (9.0): Modularize Solr by extracting obvious low hanging fruits > plugins into contribs/modules. Make it super easy to launch solr wil any of > these on class-path (SOLR-15914 > <https://issues.apache.org/jira/browse/SOLR-15914>). > Phase 2 (9.x): Evolve package manager and make it possible to optionally > install the modules as 1st party packages instead (still fat distro) > Pase 3: (10.0?): Extract even more features as modules, and publish all > modules as separate delivery artifacts on DLCDN > > Regarding phase 2 in 9.x. We cannot really extract a feature into a module > in e.g. 9.1 so users upgrading from 9.0 will get NoClassFoundException. > That breaks back-compat. But perhaps we could continue modularization > efforts in 9.x if we make sure that all new modules extracted in a minor > release are automatically added to the classloader? Then the classes will > disappear from solr-core.jar so would possibly break someone's custom > embedded usecase, but 99% of users would be unaffected. Wdyt? > > In any case, I think for 9.x the realistic route is to keep our fat tgz, > but make it slimmer by removing redundancy and prune down on the number of > overlapping dependencies. That can get us a long way. > > Jan > > 13. jan. 2022 kl. 03:15 skrev David Smiley <dsmi...@apache.org>: > > Shawn: > * RE redundancies of stuff in /dist/, see > https://issues.apache.org/jira/browse/SOLR-15916 > * RE "contrib" vs "module" vs "package", see: > https://issues.apache.org/jira/browse/SOLR-15917 > * RE not shipping these extras with the Solr distribution, see: "slim > distro" mention in the document "Solr first party packages" > https://docs.google.com/document/d/1n7gB2JAdZhlJKFrCd4Txcw4HDkdk7hlULyAZBS-wXrE/edit?usp=sharing > > It could very well be worth shipping two docker images in the meantime. > Or maybe a zip of each module could be a separate artifact that is > published? I'm not sure what freedoms we have to do this in the ASF. > > ~ David Smiley > Apache Lucene/Solr Search Developer > http://www.linkedin.com/in/davidwsmiley > > > On Wed, Jan 12, 2022 at 8:21 PM Shawn Heisey <apa...@elyograg.org> wrote: > >> On 1/12/2022 8:31 AM, Jan Høydahl wrote: >> > I think there are lots of pieces of code in solr-core that can easily >> be extracted the same way. >> > Some perhaps even for 9.0.0, as it slims down the core and reduces >> attack surface for most users as well. >> >> I think it would be really awesome if we had a core download that only >> included basic functionality, and all the other fancy things that Solr >> does now out of the box (as well as those that are contrib) could be >> added after download via package scripting or just additional downloads. >> >> The size of solr-8.11.1.tgz is 207MiB, or 218076598 bytes. The .zip >> version is slightly larger. 8.0.0 was 163MiB, 7.0.0 was 142MiBm, 6.0.0 >> was 131MiB, and 1.4.1 was 53.7MiB. I think it's insane that the >> download is so big ... and a lot of what makes it big are things that >> the vast majority of our users will never use. >> >> Large reductions in the overall size of the main download would be >> possible by putting hadoop, calcite, some of the really large lucene >> analysis components, and the contrib stuff into packages. The >> extraction contrib alone is 43.5MiB compressed in zip format. >> >> I would suggest moving zookeeper and its dependencies as well, but I >> think we probably want SolrCloud to be part of base functionality. >> >> Some of the large jars are included for what are probably insignificant >> usages, and I wonder if that functionality could be replaced by newer >> native functions available in Java 8 and later. I am eyeballing things >> like guava and the commons-* jars here, but I am sure there are other >> things in this category. I'd like to eliminate as many dependencies as >> we can. >> >> Extracting some things from the solr-core jar into other jars sounds >> like a really awesome idea. >> >> I don't think the solr-core jar should be in the dist directory. It's >> useless by itself, because it will still have a LOT of dependencies even >> if we shrink it. And there are likely other things in the dist >> directory that fall into that category. The test framework and its >> dependencies are a good candidate for removal. >> >> By removing some of the low-hanging fruit that I am SURE isn't needed >> for base binary functionality on the 8.11.1 download, I was able to end >> up with a .zip file sized in at 60.4MiB, and I am sure at least a little >> bit of further reduction is possible if we can fully map out >> dependencies. I think we can leverage gradle to provide some dependency >> info. >> >> Exactly how to organize the code repo to create divided artifacts is >> something that we would need to think about. My initial idea is >> changing "contrib" to "package" and then making some new directories >> under package. >> >> Thanks, >> Shawn >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org >> For additional commands, e-mail: dev-h...@solr.apache.org >> >> > >
