[zeromq-dev] Stable release downloads (was Re: CZMQ "stable" release versioning)
On 25/04/16 21:51, Pieter Hintjens wrote: Yes, retiring the deprecated APIs and bumping the major version should go together. It also seems timely to raise where these stable release downloads come from when a user downloads them. http://zeromq.org/intro:get-the-software links to download URLs like: http://download.zeromq.org/zeromq-4.1.4.tar.gz and download.zeromq.org is: ewen@ashram:~$ host download.zeromq.org download.zeromq.org is an alias for virtual.imatix.com. virtual.imatix.com has address 95.142.169.98 virtual.imatix.com mail is handled by 10 mail-in.imatix.com. ewen@ashram:~$ and virtual.imatix.com is an alternative name for a VM... that is paid for by iMatix (ie, by Pieter). So we'll need to find another place to host these downloads. Personally I'd like it if _all_ the existing downloads were moved to that new place, rather than just being abandoned (I've spent too much time last year/this year trying to track down old code that didn't get carried along with moves to want to cause that to happen again). That would also let us potentially make download.zeromq.org point at that new location, and thus keep existing URLs working. Currently there are 160 downloadable files on download.zeromq.org, taking up 194MB of files; so an average around 1.2MB each. With presumably some of them being downloaded more than a trivial number of times. Pieter and I briefly discussed using GitHub Releases: https://help.github.com/articles/creating-releases/ along with a custom binary which was whatever tarball, binary, etc, had been built. From what I could tell from a quick look this is plausible without cost (for an open source project), at least under 1GB total (https://help.github.com/articles/what-is-my-disk-quota/) and maybe larger if they're explicitly attached to releases (https://help.github.com/articles/distributing-large-binaries/) which then seem to get stored outside the git archive itself. But (a) there's a "technical debt" of uploading 160 files (with 1-2 files per release), and (b) there aren't even any tags on the zeromq/libzmq repository AFAICT, so there's also the technical debt of figuring out where all these files should "attach" (ie, what's the matching source). I'm explicitly _not_ volunteering to spend the time figuring out where all these tags belong, and releases should be created/uploaded.[0] But I can, eg, provide someone with a list of URLs to download all the files from download.zeromq.org if they were to volunteer to do this. There's an API to automate some of it: https://developer.github.com/v3/repos/releases/#create-a-release but that also requires a git commit ID to attach each release/set of files to... which I suspect is the hard part to figure out. Alternatively, other ideas for where to store the downloads that don't rely on iMatix/Pieter would be a good idea. Before the next release is "cut". Ewen [0] My "software archaeology" time has been going into: https://github.com/imatix-legacy/ https://imatix-legacy.github.io/ (there's still a couple of pieces to convert over to GitHub, but the imatix-legacy.github.io one is now close to replicating the early 2000 iMatix website, with the source for everything at github.com/imatix-legacy/ in at least a "human reviewable" form.) ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Stable release downloads (was Re: CZMQ "stable" release versioning)
For NetMQ we have automated release process using github releases. Once you make the release on github the CI (appvayor) compile the library and upload the artifacts to github. We can do the same with libzmq and czmq, a little bit of work though. I have less experience with travis, so I would need help with that. Regarding the existing files, I think we have two options: * Upload all to github, I can make a script that will make it simpler * Use Amazon S3, I think it will very cheap, maybe even for free, I can take care for that as well. Doron On Wed, Apr 27, 2016 at 11:52 AM, Ewen McNeill wrote: > On 25/04/16 21:51, Pieter Hintjens wrote: > >> Yes, retiring the deprecated APIs and bumping the major version should >> go together. >> > > It also seems timely to raise where these stable release downloads come > from when a user downloads them. > > http://zeromq.org/intro:get-the-software > > links to download URLs like: > > http://download.zeromq.org/zeromq-4.1.4.tar.gz > > and download.zeromq.org is: > > ewen@ashram:~$ host download.zeromq.org > download.zeromq.org is an alias for virtual.imatix.com. > virtual.imatix.com has address 95.142.169.98 > virtual.imatix.com mail is handled by 10 mail-in.imatix.com. > ewen@ashram:~$ > > and virtual.imatix.com is an alternative name for a VM... that is paid > for by iMatix (ie, by Pieter). > > So we'll need to find another place to host these downloads. Personally > I'd like it if _all_ the existing downloads were moved to that new place, > rather than just being abandoned (I've spent too much time last year/this > year trying to track down old code that didn't get carried along with moves > to want to cause that to happen again). That would also let us potentially > make download.zeromq.org point at that new location, and thus keep > existing URLs working. > > Currently there are 160 downloadable files on download.zeromq.org, taking > up 194MB of files; so an average around 1.2MB each. With presumably some > of them being downloaded more than a trivial number of times. > > Pieter and I briefly discussed using GitHub Releases: > > https://help.github.com/articles/creating-releases/ > > along with a custom binary which was whatever tarball, binary, etc, had > been built. From what I could tell from a quick look this is plausible > without cost (for an open source project), at least under 1GB total ( > https://help.github.com/articles/what-is-my-disk-quota/) and maybe larger > if they're explicitly attached to releases ( > https://help.github.com/articles/distributing-large-binaries/) which then > seem to get stored outside the git archive itself. > > But (a) there's a "technical debt" of uploading 160 files (with 1-2 files > per release), and (b) there aren't even any tags on the zeromq/libzmq > repository AFAICT, so there's also the technical debt of figuring out where > all these files should "attach" (ie, what's the matching source). > > I'm explicitly _not_ volunteering to spend the time figuring out where all > these tags belong, and releases should be created/uploaded.[0] But I can, > eg, provide someone with a list of URLs to download all the files from > download.zeromq.org if they were to volunteer to do this. > > There's an API to automate some of it: > > https://developer.github.com/v3/repos/releases/#create-a-release > > but that also requires a git commit ID to attach each release/set of files > to... which I suspect is the hard part to figure out. > > Alternatively, other ideas for where to store the downloads that don't > rely on iMatix/Pieter would be a good idea. Before the next release is > "cut". > > Ewen > > [0] My "software archaeology" time has been going into: > > https://github.com/imatix-legacy/ > https://imatix-legacy.github.io/ > > (there's still a couple of pieces to convert over to GitHub, but the > imatix-legacy.github.io one is now close to replicating the early 2000 > iMatix website, with the source for everything at > github.com/imatix-legacy/ in at least a "human reviewable" form.) > ___ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org > http://lists.zeromq.org/mailman/listinfo/zeromq-dev > ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Stable release downloads (was Re: CZMQ "stable" release versioning)
On Wed, Apr 27, 2016 at 4:23 PM, Doron Somech wrote: > For NetMQ we have automated release process using github releases. Once you > make the release on github the CI (appvayor) compile the library and upload > the artifacts to github. > > We can do the same with libzmq and czmq, a little bit of work though. I have > less experience with travis, so I would need help with that. > > Regarding the existing files, I think we have two options: > * Upload all to github, I can make a script that will make it simpler > * Use Amazon S3, I think it will very cheap, maybe even for free, I can take > care for that as well. Github is still a pain, and it does not replaces the simplicity of a static http/ftp server: http://www.zoobab.com/github-sucks-wget-unfriendly For example, if you want to make a mirror of all tarball releases, it is a nightmare. -- Benjamin Henrion FFII Brussels - +32-484-566109 - +32-2-3500762 "In July 2005, after several failed attempts to legalise software patents in Europe, the patent establishment changed its strategy. Instead of explicitly seeking to sanction the patentability of software, they are now seeking to create a central European patent court, which would establish and enforce patentability rules in their favor, without any possibility of correction by competing courts or democratically elected legislators." ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Stable release downloads (was Re: CZMQ "stable" release versioning)
On 28/04/16 2:23, Doron Somech wrote: Regarding the existing files, I think we have two options: * Upload all to github, I can make a script that will make it simpler * Use Amazon S3, I think it will very cheap, maybe even for free, I can take care for that as well. AFAICT, uploading the binary release artefacts to GitHub, results in GitHub storing them on S3 (at GitHub's expense). Given the source repositories are on GitHub, I think that'd be my default choice for the existing files. Thanks for the offer to help with scripting uploads of the existing files. I think the main "hard" bit (ie, require a human) left would be figuring out which commits to put the release tags on, so that they match up with what is in the tarballs. Maybe someone else can help with identifying those commits? Directly using S3 is worth considering, but I think at best it would be free for 1 year: https://aws.amazon.com/free/faqs/ "5 GB of Amazon S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests*" has the little star that means "available for 12 months following your AWS sign-up date'. So going the S3 route would mean finding some way to fund it on an on-going basis. It's relatively cheap (https://aws.amazon.com/s3/pricing/), but not free ($0.03/GB for storage/month, plus $0.09/GB for transfers to the Internet; at a guess less than a few dollars a month, but having to be paid every month). Ewen PS: Re http://www.zoobab.com/github-sucks-wget-unfriendly, the issue seems to be the URL structure doesn't have the filename at the end. Annoying, undesirable, but IMHO, not a fatal reason not to use it. It'd perhaps be better to lobby GitHub to change their URL structure to end in the original filename and/or a simulated final filename; it should be a relatively simple change for them. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Stable release downloads (was Re: CZMQ "stable" release versioning)
On Wed, Apr 27, 2016 at 10:40 PM, Ewen McNeill wrote: > On 28/04/16 2:23, Doron Somech wrote: >> >> Regarding the existing files, I think we have two options: >> * Upload all to github, I can make a script that will make it simpler >> * Use Amazon S3, I think it will very cheap, maybe even for free, I can >> take care for that as well. > > > AFAICT, uploading the binary release artefacts to GitHub, results in GitHub > storing them on S3 (at GitHub's expense). Given the source repositories are > on GitHub, I think that'd be my default choice for the existing files. > > Thanks for the offer to help with scripting uploads of the existing files. > I think the main "hard" bit (ie, require a human) left would be figuring out > which commits to put the release tags on, so that they match up with what is > in the tarballs. Maybe someone else can help with identifying those > commits? > > Directly using S3 is worth considering, but I think at best it would be free > for 1 year: > > https://aws.amazon.com/free/faqs/ > > "5 GB of Amazon S3 standard storage, 20,000 Get Requests, and 2,000 Put > Requests*" > > has the little star that means "available for 12 months following your AWS > sign-up date'. So going the S3 route would mean finding some way to fund it > on an on-going basis. It's relatively cheap > (https://aws.amazon.com/s3/pricing/), but not free ($0.03/GB for > storage/month, plus $0.09/GB for transfers to the Internet; at a guess less > than a few dollars a month, but having to be paid every month). > > Ewen > > PS: Re http://www.zoobab.com/github-sucks-wget-unfriendly, the issue seems > to be the URL structure doesn't have the filename at the end. Annoying, > undesirable, but IMHO, not a fatal reason not to use it. It'd perhaps be > better to lobby GitHub to change their URL structure to end in the original > filename and/or a simulated final filename; it should be a relatively simple > change for them. Let's prepare an open letter. I am sure lot of people will sign it up. -- Benjamin Henrion FFII Brussels - +32-484-566109 - +32-2-3500762 "In July 2005, after several failed attempts to legalise software patents in Europe, the patent establishment changed its strategy. Instead of explicitly seeking to sanction the patentability of software, they are now seeking to create a central European patent court, which would establish and enforce patentability rules in their favor, without any possibility of correction by competing courts or democratically elected legislators." ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
Re: [zeromq-dev] Stable release downloads (was Re: CZMQ "stable" release versioning)
On 28/04/16 19:56, Benjamin Henrion wrote: PS: Re http://www.zoobab.com/github-sucks-wget-unfriendly, the issue seems to be the URL structure doesn't have the filename at the end. Let's prepare an open letter. I am sure lot of people will sign it up. From some digging around there appears to be an _unofficial_ de facto "github issues" tracker here: https://github.com/isaacs/github/issues (because GitHub don't have one of their own); it seems to be getting "up votes" and the like from people. There's about 500 issues there neatly tagged with things like "releases", "enhancement", etc. Official requests for GitHub itself to actually do something seem to be still via supp...@github.com. So possibly the best idea is to add an issue to that unofficial github issues tracker and then email supp...@github.com with (a) the text of the issue (so it's right in front of them) and (b) the URL to the issue (so they can see other people who agree it's an issue). Then encourage people to comment on that issue if it bugs them. I think we want to ask something like "please add 'file/FILENAME' to the end of your release URLs, so that software that uses the last component of the filename when saving the file, eg, wget, will automatically save the release with the right filename." Is that consistent with your understanding of the issue? Ewen PS: There are a few more "issues" in there about the release feature, but AFAICT on a quick glance none of them are specifically for that particular URL structure issue. ___ zeromq-dev mailing list zeromq-dev@lists.zeromq.org http://lists.zeromq.org/mailman/listinfo/zeromq-dev
