Considering the are no updates, why would we even serve the nbms? To older 8.2 installs or what? Won't those older installs request the 10 nbms anyhow? So, I don't see how those nbms would get so many downloads.
We can put the smaller updates.xml.gz anywhere, it won't make any impact on bandwidth. I would not use any 3rd party solution unless Apache infra tell us so. The one thing Apache provides to projects is infrastructure. I say we pick the simplest solution: use archive.apache.org for everything. It will serve only the updates.xml.gz in reality and if they do get too much traffic infra will let us know and we figure something out then. BTW, I still believe we have to create, sign and store the nbms, even if they are only convenience binaries. --emi http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more! On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi <laszlo.kisha...@gmail.com> wrote: > > Dear all, > > I have to bring up this old item again. According to the Apache release > processes, I'd need to archive the current 9.0 release from the site. > this means that our 9.0 NBM-s are going to be removed from there as > well. The files are going to be available on archive.apache.org which is > not mirrored. Only the most recent release are supposed to be out on the > release and their mirrors. > > This means we need to find some place to our 9.0 nbms. I also won't mind > if we could move away storing our nbms, "officially" (there are no > official binary releases, yet) on some other place than Apache dist > infrastructure. It's a bit weird to sign and upload 650+ binaries with > each and every release. So let's go back to square one. > > 1. We can change the redirection to the Apache archive site. That is > not mirrored, and initially we would put a real traffic on there. > 2. We can serve the nbm-s from our netbeans-vm. > Checked the stats we get ~25000 requests a month on updates.xml.gz > if we'd transfer all the update center (~120 Mb, for NetBeans 10.0) > all the modules with each request for updates.xml.gz our transfer > would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month > usage limit. That would be actually much less, as this is a worst > case scenario. > 3. Search for a free third party solution. > I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty > reliable, the only concern is the 1 Tb of network limit per month. > Unfortunately I only can guess that the 3 Tb is really exaggerated. > All the log files I've found do not include the actual download > statistics of our nbm-s > 4. Something else? > > I can't archive the current 9.0 till this thing is decided. > > On 4/10/18 11:39 PM, Antonio wrote: > > Hi all, > > > > In order to understand this long thread (about NETBEANS-330 [1]) I > > tried to summarize it. Please review and send corrections & questions > > as appropriate. > > > > Kind regards, > > Antonio > > > > == Objectives > > > > Host the NetBeans 9.0 Update Center on Apache infrastructure. > > > > == Constraints/facts/options > > > > - We cannot host it on a website due to bandwidth requirements of 3-5 > > Tb/month. > > > > - We must use the Apache Mirror system and their "closer.cgi, > > closer.lua" cgi scripts [2] to select the closest mirror to the user. > > This script can either redirect directly to the closest binary file > > (ready to download), or return a JSON response with a list of closest > > mirrors. > > > > - Distributing through the Apache Mirror system requires a proper > > release, with voting, approval and signing, etc. > > > > - Geertjan says that prior to the final release we could do an rc > > release to have this feature tested. > > > > - The UC catalog xml file can be hosted in the mirror system. For this > > to work the catalog xml file must contain relative paths, so when aged > > releases are moved away from the mirror system and into the archive > > things keep working. > > > > - We can HTTP redirect to the proper catalog.xml file... > > > > a) ... From our website, redirecting to the Apache mirror system or to > > the archive with an .htaccess file (under our control). > > > > b) ... Idem, by asking infra to modify our server configuration (more > > performant but requires Infra tickets). > > > > c) ... Using a script of ours hosted at the "VM" (a web server ours > > currently hosting selenium), that may also track/log some statistics. > > > > - The NetBeans UC module will make a request for the XML file to our > > website, from there will be redirected to the mirror/archive. Later on > > it will build up proper URLs (using some prefix and the relative paths > > inside the catalog xml file) for the final NBM downloads. > > > > - Mirror download statistics may be currently downloaded from > > https://www.apache.org/dyn/stats/netbeans.log > > > > - Web server statistics are available through a very detailed request > > to infra. They say they're not in the "counting business" :-) > > > > - Jan is working in NBM generation (with NOTICE & LICENSE, etc.). > > Jenkins now generates these artifacts: > > https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/ > > > > > > == Open issues & questions > > > > 1- It seems we won't be able to host some OSGi bundles in the mirror > > system due to licensing issues? > > > > 2- Mirrors are not as reliable as a CDN: files can be corrupt, or > > whatever. The UC module should verify the integrity of the downloaded > > files, etc. > > > > 3- We may want to select a small set of NBM files for the first run, > > to workaround previous licensing problems. > > > > > > [1] > > https://issues.apache.org/jira/browse/NETBEANS-330 > > > > [2] > > https://reference.apache.org/pmc/mirror_scripts > > > > > > On 05/04/18 14:38, Geertjan Wielenga wrote: > >> Hi all, > >> > >> We need to nail down this one and I think the key blocker is that > >> there are > >> different ideas about what this is about: > >> > >> https://issues.apache.org/jira/browse/NETBEANS-330 > >> > >> The above is not about the Plugin Portal. > >> > >> If I understand it correctly, this is about where the NBMs (which > >> ones? how > >> many? do we know?) and the related XML file (a.k.a. update center) > >> will be > >> hosted. > >> > >> AFAIK, the XML file and the NBMs could be put onto our Apache > >> NetBeans VM > >> just like Synergy: > >> > >> http://netbeans-vm.apache.org/synergy > >> > >> The key question remains, which NBMs are we talking about here, > >> applicable > >> to the 9.0 release, I think. > >> > >> Thanks, > >> > >> Gj > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org > > For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org > > > > For further information about the NetBeans mailing lists, visit: > > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
