Hi all,
I'm asking at infra.chat if we should use the archive or
netbeans-vm.apache.org.
Cheers,
Antonio
El 27/12/2018 a las 9:46, Emilian Bold escribió:
That's no questions. My only concern is where to store.
We store on Apache infra and if it's too much for them we ask them for
suggestions or new servers.
I understand how you computed but for older nbms I don't believe we
will have even a fraction of those maximum downloads.
--emi
http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
On Thu, Dec 27, 2018 at 10:44 AM Laszlo Kishalmi
<laszlo.kisha...@gmail.com> wrote:
On 12/26/18 11:23 PM, Emilian Bold wrote:
Considering the are no updates, why would we even serve the nbms? To
older 8.2 installs or what? Won't those older installs request the 10
nbms anyhow? So, I don't see how those nbms would get so many
downloads.
I've just really calculated the worst case scenario for the download
traffic:
1. Number of request per month on updates.xml.gz is 22 - 25 thousand I
took 25000
2. NetBeans 10.0 nbm-s take more space (118Mb) than 9.0, so I
calculated with 120Mb
So if everyone would download the whole nbm structure whenever it hits
the updates.xml.gz, then we could have 25000 * 120 Mb = 3 Tb per month
download traffic.
We can put the smaller updates.xml.gz anywhere, it won't make any
impact on bandwidth.
I would not use any 3rd party solution unless Apache infra tell us so.
The one thing Apache provides to projects is infrastructure.
Well, Apache provides more than infra, however that infra not necessary
designed for a project like us., that are producing 650+ artifacts,
weighs ~500 Mb with every release.
I say we pick the simplest solution: use archive.apache.org for
everything. It will serve only the updates.xml.gz in reality and if
they do get too much traffic infra will let us know and we figure
something out then.
Well, that's might be the easiest solution, still option 2 or 3 would be
easy to implement as well and clean up our project layout on
dist.apache.org.
BTW, I still believe we have to create, sign and store the nbms, even
if they are only convenience binaries.
That's no questions. My only concern is where to store.
--emi
http://coolbeans.xyz/ - CoolBeans: An IDE for Java, JavaEE, PHP and more!
On Thu, Dec 27, 2018 at 8:11 AM Laszlo Kishalmi
<laszlo.kisha...@gmail.com> wrote:
Dear all,
I have to bring up this old item again. According to the Apache release
processes, I'd need to archive the current 9.0 release from the site.
this means that our 9.0 NBM-s are going to be removed from there as
well. The files are going to be available on archive.apache.org which is
not mirrored. Only the most recent release are supposed to be out on the
release and their mirrors.
This means we need to find some place to our 9.0 nbms. I also won't mind
if we could move away storing our nbms, "officially" (there are no
official binary releases, yet) on some other place than Apache dist
infrastructure. It's a bit weird to sign and upload 650+ binaries with
each and every release. So let's go back to square one.
1. We can change the redirection to the Apache archive site. That is
not mirrored, and initially we would put a real traffic on there.
2. We can serve the nbm-s from our netbeans-vm.
Checked the stats we get ~25000 requests a month on updates.xml.gz
if we'd transfer all the update center (~120 Mb, for NetBeans 10.0)
all the modules with each request for updates.xml.gz our transfer
would be ~3 Tb. So we would definitely fit into the 3-5 Tb/per month
usage limit. That would be actually much less, as this is a worst
case scenario.
3. Search for a free third party solution.
I'd try Bintray, we can get 10 Gb space and CDN, so it is pretty
reliable, the only concern is the 1 Tb of network limit per month.
Unfortunately I only can guess that the 3 Tb is really exaggerated.
All the log files I've found do not include the actual download
statistics of our nbm-s
4. Something else?
I can't archive the current 9.0 till this thing is decided.
On 4/10/18 11:39 PM, Antonio wrote:
Hi all,
In order to understand this long thread (about NETBEANS-330 [1]) I
tried to summarize it. Please review and send corrections & questions
as appropriate.
Kind regards,
Antonio
== Objectives
Host the NetBeans 9.0 Update Center on Apache infrastructure.
== Constraints/facts/options
- We cannot host it on a website due to bandwidth requirements of 3-5
Tb/month.
- We must use the Apache Mirror system and their "closer.cgi,
closer.lua" cgi scripts [2] to select the closest mirror to the user.
This script can either redirect directly to the closest binary file
(ready to download), or return a JSON response with a list of closest
mirrors.
- Distributing through the Apache Mirror system requires a proper
release, with voting, approval and signing, etc.
- Geertjan says that prior to the final release we could do an rc
release to have this feature tested.
- The UC catalog xml file can be hosted in the mirror system. For this
to work the catalog xml file must contain relative paths, so when aged
releases are moved away from the mirror system and into the archive
things keep working.
- We can HTTP redirect to the proper catalog.xml file...
a) ... From our website, redirecting to the Apache mirror system or to
the archive with an .htaccess file (under our control).
b) ... Idem, by asking infra to modify our server configuration (more
performant but requires Infra tickets).
c) ... Using a script of ours hosted at the "VM" (a web server ours
currently hosting selenium), that may also track/log some statistics.
- The NetBeans UC module will make a request for the XML file to our
website, from there will be redirected to the mirror/archive. Later on
it will build up proper URLs (using some prefix and the relative paths
inside the catalog xml file) for the final NBM downloads.
- Mirror download statistics may be currently downloaded from
https://www.apache.org/dyn/stats/netbeans.log
- Web server statistics are available through a very detailed request
to infra. They say they're not in the "counting business" :-)
- Jan is working in NBM generation (with NOTICE & LICENSE, etc.).
Jenkins now generates these artifacts:
https://builds.apache.org/view/Incubator%20Projects/job/incubator-netbeans-linux/lastSuccessfulBuild/artifact/nbbuild/nbms/
== Open issues & questions
1- It seems we won't be able to host some OSGi bundles in the mirror
system due to licensing issues?
2- Mirrors are not as reliable as a CDN: files can be corrupt, or
whatever. The UC module should verify the integrity of the downloaded
files, etc.
3- We may want to select a small set of NBM files for the first run,
to workaround previous licensing problems.
[1]
https://issues.apache.org/jira/browse/NETBEANS-330
[2]
https://reference.apache.org/pmc/mirror_scripts
On 05/04/18 14:38, Geertjan Wielenga wrote:
Hi all,
We need to nail down this one and I think the key blocker is that
there are
different ideas about what this is about:
https://issues.apache.org/jira/browse/NETBEANS-330
The above is not about the Plugin Portal.
If I understand it correctly, this is about where the NBMs (which
ones? how
many? do we know?) and the related XML file (a.k.a. update center)
will be
hosted.
AFAIK, the XML file and the NBMs could be put onto our Apache
NetBeans VM
just like Synergy:
http://netbeans-vm.apache.org/synergy
The key question remains, which NBMs are we talking about here,
applicable
to the 9.0 release, I think.
Thanks,
Gj
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org
For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org
For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org
For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@netbeans.incubator.apache.org
For additional commands, e-mail: dev-h...@netbeans.incubator.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
