> That's no questions. My only concern is where to store. We store on Apache infra and if it's too much for them we ask them for suggestions or new servers.
I understand how you computed but for older nbms I don't believe we will have even a fraction of those maximum downloads. --emi http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more! On Thu, Dec 27, 2018 at 10:44 AM Laszlo Kishalmi <laszlo.kisha...@gmail.com> wrote: > > On 12/26/18 11:23 PM, Emilian Bold wrote: > > Considering the are no updates, why would we even serve the nbms? To > > older 8.2 installs or what? Won't those older installs request the 10 > > nbms anyhow? So, I don't see how those nbms would get so many > > downloads. > > I've just really calculated the worst case scenario for the download > traffic: > > 1. Number of request per month on updates.xml.gz is 22 - 25 thousand I > took 25000 > 2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I > calculated with 120Mb > > So if everyone would download the whole nbm structure whenever it hits > the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month > download traffic. > > > We can put the smaller updates.xml.gz anywhere, it won't make any > > impact on bandwidth. > > > > I would not use any 3rd party solution unless Apache infra tell us so. > > The one thing Apache provides to projects is infrastructure. > Well, Apache provides more than infra, however that infra not necessary > designed for a project like us., that are producing 650+ artifacts, > weighs ~500 Mb with every release. > > > > I say we pick the simplest solution: use archive.apache.org for > > everything. It will serve only the updates.xml.gz in reality and if > > they do get too much traffic infra will let us know and we figure > > something out then. > Well, that's might be the easiest solution, still option 2 or 3 would be > easy to implement as well and clean up our project layout on > dist.apache.org. > > > > BTW, I still believe we have to create, sign and store the nbms, even > > if they are only convenience binaries. > That's no questions. My only concern is where to store. > > > > --emi > > > > http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more! > > > > On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi > > <laszlo.kisha...@gmail.com> wrote: > >> Dear all, > >> > >> I have to bring up this old item again. According to the Apache release > >> processes, I'd need to archive the current 9.0 release from the site. > >> this means that our 9.0 NBM-s are going to be removed from there as > >> well. The files are going to be available on archive.apache.org which is > >> not mirrored. Only the most recent release are supposed to be out on the > >> release and their mirrors. > >> > >> This means we need to find some place to our 9.0 nbms. I also won't mind > >> if we could move away storing our nbms, "officially" (there are no > >> official binary releases, yet) on some other place than Apache dist > >> infrastructure. It's a bit weird to sign and upload 650+ binaries with > >> each and every release. So let's go back to square one. > >> > >> 1. We can change the redirection to the Apache archive site. That is > >> not mirrored, and initially we would put a real traffic on there. > >> 2. We can serve the nbm-s from our netbeans-vm. > >> Checked the stats we get ~25000 requests a month on updates.xml.gz > >> if we'd transfer all the update center (~120 Mb, for NetBeans 10.0) > >> all the modules with each request for updates.xml.gz our transfer > >> would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month > >> usage limit. That would be actually much less, as this is a worst > >> case scenario. > >> 3. Search for a free third party solution. > >> I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty > >> reliable, the only concern is the 1 Tb of network limit per month. > >> Unfortunately I only can guess that the 3 Tb is really exaggerated. > >> All the log files I've found do not include the actual download > >> statistics of our nbm-s > >> 4. Something else? > >> > >> I can't archive the current 9.0 till this thing is decided. > >> > >> On 4/10/18 11:39 PM, Antonio wrote: > >>> Hi all, > >>> > >>> In order to understand this long thread (about NETBEANS-330 [1]) I > >>> tried to summarize it. Please review and send corrections & questions > >>> as appropriate. > >>> > >>> Kind regards, > >>> Antonio > >>> > >>> == Objectives > >>> > >>> Host the NetBeans 9.0 Update Center on Apache infrastructure. > >>> > >>> == Constraints/facts/options > >>> > >>> - We cannot host it on a website due to bandwidth requirements of 3-5 > >>> Tb/month. > >>> > >>> - We must use the Apache Mirror system and their "closer.cgi, > >>> closer.lua" cgi scripts [2] to select the closest mirror to the user. > >>> This script can either redirect directly to the closest binary file > >>> (ready to download), or return a JSON response with a list of closest > >>> mirrors. > >>> > >>> - Distributing through the Apache Mirror system requires a proper > >>> release, with voting, approval and signing, etc. > >>> > >>> - Geertjan says that prior to the final release we could do an rc > >>> release to have this feature tested. > >>> > >>> - The UC catalog xml file can be hosted in the mirror system. For this > >>> to work the catalog xml file must contain relative paths, so when aged > >>> releases are moved away from the mirror system and into the archive > >>> things keep working. > >>> > >>> - We can HTTP redirect to the proper catalog.xml file... > >>> > >>> a) ... From our website, redirecting to the Apache mirror system or to > >>> the archive with an .htaccess file (under our control). > >>> > >>> b) ... Idem, by asking infra to modify our server configuration (more > >>> performant but requires Infra tickets). > >>> > >>> c) ... Using a script of ours hosted at the "VM" (a web server ours > >>> currently hosting selenium), that may also track/log some statistics. > >>> > >>> - The NetBeans UC module will make a request for the XML file to our > >>> website, from there will be redirected to the mirror/archive. Later on > >>> it will build up proper URLs (using some prefix and the relative paths > >>> inside the catalog xml file) for the final NBM downloads. > >>> > >>> - Mirror download statistics may be currently downloaded from > >>> https://www.apache.org/dyn/stats/netbeans.log > >>> > >>> - Web server statistics are available through a very detailed request > >>> to infra. They say they're not in the "counting business" :-) > >>> > >>> - Jan is working in NBM generation (with NOTICE & LICENSE, etc.). > >>> Jenkins now generates these artifacts: > >>> https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/ > >>> > >>> > >>> == Open issues & questions > >>> > >>> 1- It seems we won't be able to host some OSGi bundles in the mirror > >>> system due to licensing issues? > >>> > >>> 2- Mirrors are not as reliable as a CDN: files can be corrupt, or > >>> whatever. The UC module should verify the integrity of the downloaded > >>> files, etc. > >>> > >>> 3- We may want to select a small set of NBM files for the first run, > >>> to workaround previous licensing problems. > >>> > >>> > >>> [1] > >>> https://issues.apache.org/jira/browse/NETBEANS-330 > >>> > >>> [2] > >>> https://reference.apache.org/pmc/mirror_scripts > >>> > >>> > >>> On 05/04/18 14:38, Geertjan Wielenga wrote: > >>>> Hi all, > >>>> > >>>> We need to nail down this one and I think the key blocker is that > >>>> there are > >>>> different ideas about what this is about: > >>>> > >>>> https://issues.apache.org/jira/browse/NETBEANS-330 > >>>> > >>>> The above is not about the Plugin Portal. > >>>> > >>>> If I understand it correctly, this is about where the NBMs (which > >>>> ones? how > >>>> many? do we know?) and the related XML file (a.k.a. update center) > >>>> will be > >>>> hosted. > >>>> > >>>> AFAIK, the XML file and the NBMs could be put onto our Apache > >>>> NetBeans VM > >>>> just like Synergy: > >>>> > >>>> http://netbeans-vm.apache.org/synergy > >>>> > >>>> The key question remains, which NBMs are we talking about here, > >>>> applicable > >>>> to the 9.0 release, I think. > >>>> > >>>> Thanks, > >>>> > >>>> Gj > >>>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org > >>> For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org > >>> > >>> For further information about the NetBeans mailing lists, visit: > >>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > >>> > >>> > >>> > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org > > For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org > > > > For further information about the NetBeans mailing lists, visit: > > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org For further information about the NetBeans mailing lists, visit: https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
