Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
On 2018-08-19 03:09 PM, Daniel F. Dickinson wrote: > Since it's holiday season I will wait until after first full week of > September to close https://github.com/openwrt/packages/issues/6748 which > discusses this issue. > > I would like to say that impatience (and I'm badly guilty of this too), > is a big part of the problem here and with the perceived > maintainer problem with the packages feed, and/or the pushing of commits > by non-maintainers. Also due to objections with using the list for this discussion (and most posting being on the issue) I'd suggest to move the discussion to the ticket. I admit I'm not particularly happy about the lack of a proper discussion list for the packages feed (I don't consider GitHub Issues very helpful in that regard although apparently others do). Regards, Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
Since it's holiday season I will wait until after first full week of September to close https://github.com/openwrt/packages/issues/6748 which discusses this issue. I would like to say that impatience (and I'm badly guilty of this too), is a big part of the problem here and with the perceived maintainer problem with the packages feed, and/or the pushing of commits by non-maintainers. Regards, Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
On 2018-08-13 01:45 PM, Rosen Penev wrote: > On Mon, Aug 13, 2018 at 2:45 AM Jo-Philipp Wich wrote: >> Hi, >> >> personally I'm opposed to the entire code load thing. >> >> First of all I was unable to reproduce the tarballs offered by Github. >> >> Github seems to use an extended tar (pax) format while we pack our SCM >> clones using the more traditional ustar format, however even using `tar >> -cp -H pax --numeric-owner --owner=0 --group=0 --sort=name --mtime ...` >> seems to yield a different tar stream compared to whatever is offered by >> Github; >> >> - The order of the entries in the archive also seems to deviate from >>that of `tar --sort=name`, it looks as if Github archives are sorted >>using the "C" collate while GNU tar uses something else. >> >> - The PAX header format seems to be different, Github uses a global PAX >>header while GNU tar produces per-member headers >> >> - There seem to be proprietary tags inside Github tar (comment=) >>which are not present in the GNU equivalent >> >> Furthermore I dislike the idea of tailoring download mechanisms around a >> specific proprietary service. >> >> If the allegations about hash changes for unknown reasons are correct, >> then this raises a huge red flag for me > I can only imagine this to be the case if git history gets changed. > I'm sure it's automated. >> and I see no reason to not >> assume that codeload tarballs will eventually change as well, become >> rate limited, redirected, discontinued or changed in other arbitrary ways. > Well, there are people who believe Microsoft will cripple GitHub. >> So TLDR; I prefer a locally reproducible, cached tarball of a given SCM >> clone over an opaque Github offer. > It's not reproducible in all cases. See > https://github.com/openwrt/openwrt/pull/815 > > We could not produce the same tarball with the same hash. I was using > Ubuntu 16.04 FWIW. >> >> My 2cents, > Mine are that it's better to use a tarball that is not locally > generated. I understand that ultimately the buildbots generate the > tarballs but then you potentially have a situation where a package > bump has the wrong hash since the buildbot and someone's local > environment can differ enough to matter. > > My understanding is that OpenWrt can be built on Linux and some BSDs > like macOS or FreeBSD. > > The initial motivation for moving packages to codeload was to adjust > uscan's reporting of version numbers as git revisions when there are > git tags available. eg: using > > PKG_SOURCE_VERSION:=1ee4c4eac2f1f771ecc57d4f7ce298739bbc8a82 vs. > PKG_SOURCE_VERSION:=v$(PKG_VERSION) > > I also don't see why both codeload and buildbot generated tarballs > can't coexist. Based on actual responses (most of which were on the GitHub issue not here), it seems the majority opinion is to not actually care about this / leave it individual maintainers to do decide. If that's not the case please post on the ticket and/or here else I shall close ticket and Rosen Perev and Daniel Englberg(sp?) (@neheb and @diizzy) will be promoting this to maintainers on GitHub. I wanted to make sure this was discussed openly, but if not enough people actually care to comment, and it seems majority opinion is to no care, then I guess the discussion wasn't needed. ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
On 2018-08-13 01:45 PM, Rosen Penev wrote: >> Furthermore I dislike the idea of tailoring download mechanisms around a >> specific proprietary service. This I agree with >> >> If the allegations about hash changes for unknown reasons are correct, >> then this raises a huge red flag for me > I can only imagine this to be the case if git history gets changed. > I'm sure it's automated. Well it's been happening intermittently for a least 2-3 years from what I've seen myself and heard from others. My own theory is a flaw in their reproducible builds strategy, and for some reason or other automated rebuilding of the tarballs even when the code hasn't changed. >> and I see no reason to not >> assume that codeload tarballs will eventually change as well, become >> rate limited, redirected, discontinued or changed in other arbitrary ways. > Well, there are people who believe Microsoft will cripple GitHub. I don't MS is relevant to the question, only profit motive, TBH. It's more have a reasonable concern that the drive to monetize will result in changes that make community more difficult (at best). >> >> So TLDR; I prefer a locally reproducible, cached tarball of a given SCM >> clone over an opaque Github offer. > It's not reproducible in all cases. See > https://github.com/openwrt/openwrt/pull/815 > > We could not produce the same tarball with the same hash. I was using > Ubuntu 16.04 FWIW Can you reproduce the non-reproduction, or was it a one-off that could be attribute temporary local issues? I'm sure the OpenWrt core team would really like to know about verifiable cases of issues with failures to reproduce, since this is considered rather important. >> >> >> My 2cents, > Mine are that it's better to use a tarball that is not locally > generated. I understand that ultimately the buildbots generate the > tarballs but then you potentially have a situation where a package > bump has the wrong hash since the buildbot and someone's local > environment can differ enough to matter. The whole point of the drive for reproducible builds (and tarballs) is to identify all factors that influence the final hash and so that one can be guaranteed of the same result irrespective of environment. If there is verifiable evidence somethings been missed, please file details at https://bugs.openwrt.org as that's not just an issue with an individual package but with the core system. > > My understanding is that OpenWrt can be built on Linux and some BSDs > like macOS or FreeBSD. > > The initial motivation for moving packages to codeload was to adjust > uscan's reporting of version numbers as git revisions when there are > git tags available. eg: using > > PKG_SOURCE_VERSION:=1ee4c4eac2f1f771ecc57d4f7ce298739bbc8a82 vs. > PKG_SOURCE_VERSION:=v$(PKG_VERSION) I'm not exactly convinced that's a good reason to adopt a tree-wide change involving a propriety and opaque mechanism. > > I also don't see why both codeload and buildbot generated tarballs > can't coexist. Have you looked at how the build system works in this regard? Regards, Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
On Mon, Aug 13, 2018 at 2:45 AM Jo-Philipp Wich wrote: > > Hi, > > personally I'm opposed to the entire code load thing. > > First of all I was unable to reproduce the tarballs offered by Github. > > Github seems to use an extended tar (pax) format while we pack our SCM > clones using the more traditional ustar format, however even using `tar > -cp -H pax --numeric-owner --owner=0 --group=0 --sort=name --mtime ...` > seems to yield a different tar stream compared to whatever is offered by > Github; > > - The order of the entries in the archive also seems to deviate from >that of `tar --sort=name`, it looks as if Github archives are sorted >using the "C" collate while GNU tar uses something else. > > - The PAX header format seems to be different, Github uses a global PAX >header while GNU tar produces per-member headers > > - There seem to be proprietary tags inside Github tar (comment=) >which are not present in the GNU equivalent > > Furthermore I dislike the idea of tailoring download mechanisms around a > specific proprietary service. > > If the allegations about hash changes for unknown reasons are correct, > then this raises a huge red flag for me I can only imagine this to be the case if git history gets changed. I'm sure it's automated. > and I see no reason to not > assume that codeload tarballs will eventually change as well, become > rate limited, redirected, discontinued or changed in other arbitrary ways. Well, there are people who believe Microsoft will cripple GitHub. > > So TLDR; I prefer a locally reproducible, cached tarball of a given SCM > clone over an opaque Github offer. It's not reproducible in all cases. See https://github.com/openwrt/openwrt/pull/815 We could not produce the same tarball with the same hash. I was using Ubuntu 16.04 FWIW. > > > My 2cents, Mine are that it's better to use a tarball that is not locally generated. I understand that ultimately the buildbots generate the tarballs but then you potentially have a situation where a package bump has the wrong hash since the buildbot and someone's local environment can differ enough to matter. My understanding is that OpenWrt can be built on Linux and some BSDs like macOS or FreeBSD. The initial motivation for moving packages to codeload was to adjust uscan's reporting of version numbers as git revisions when there are git tags available. eg: using PKG_SOURCE_VERSION:=1ee4c4eac2f1f771ecc57d4f7ce298739bbc8a82 vs. PKG_SOURCE_VERSION:=v$(PKG_VERSION) I also don't see why both codeload and buildbot generated tarballs can't coexist. > Jo > > ___ > openwrt-devel mailing list > openwrt-devel@lists.openwrt.org > https://lists.openwrt.org/mailman/listinfo/openwrt-devel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
On 13/08/18 11:45, Jo-Philipp Wich wrote: So TLDR; I prefer a locally reproducible, cached tarball of a given SCM clone over an opaque Github offer. I would like to second that notion John ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
Re: [OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
Hi, personally I'm opposed to the entire code load thing. First of all I was unable to reproduce the tarballs offered by Github. Github seems to use an extended tar (pax) format while we pack our SCM clones using the more traditional ustar format, however even using `tar -cp -H pax --numeric-owner --owner=0 --group=0 --sort=name --mtime ...` seems to yield a different tar stream compared to whatever is offered by Github; - The order of the entries in the archive also seems to deviate from that of `tar --sort=name`, it looks as if Github archives are sorted using the "C" collate while GNU tar uses something else. - The PAX header format seems to be different, Github uses a global PAX header while GNU tar produces per-member headers - There seem to be proprietary tags inside Github tar (comment=) which are not present in the GNU equivalent Furthermore I dislike the idea of tailoring download mechanisms around a specific proprietary service. If the allegations about hash changes for unknown reasons are correct, then this raises a huge red flag for me and I see no reason to not assume that codeload tarballs will eventually change as well, become rate limited, redirected, discontinued or changed in other arbitrary ways. So TLDR; I prefer a locally reproducible, cached tarball of a given SCM clone over an opaque Github offer. My 2cents, Jo ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
[OpenWrt-Devel] openwrt/packages: [RFC] Regarding preferences re: switch to codeload
Posting here as this seems to me to be a project decision: From: https://github.com/openwrt/packages/issues/6748 > @neheb Has submitted a number of PR's (referenced in #6584), full list of > his/her PR's (which probably include the codeload thing for any github > pulls): https://github.com/openwrt/packages/pulls/neheb > > @diizzyy has also mentioned codeload. @karlp appears to feel the change is > for marginal gain (based on #6584 (comment)). > > I personally think that tree-wide changes, especially for something @yousong > has been trying solve another way ought t be clearly discussed and not just > brought in through the back door, hence this ticket and I'll be following up > on openwrt-devel as well. ** Clarification for neheb this is for GitHub projects without release tarballs; of course as I understand the GitHub release tarball situation, the hash changes for unknown reasons for time to time, which makes them useless. Regards, Daniel ___ openwrt-devel mailing list openwrt-devel@lists.openwrt.org https://lists.openwrt.org/mailman/listinfo/openwrt-devel
