Re: scratch buildds
On 15.06.19 11:20, Helmut Grohne wrote:
Hi,
> Unlike the ${dist}-proposed variant, the scratch distribution can be set> up
> entirely outside Debian. It only needs someone doing the work with
no> involvement of DSA. Wait, this reminds me of something. Luca
Falavigna> put up debomatic-${arch}.debian.net. And it has piuparts and
lintian!
As somebody who does backports stuff and project/client specific repos,
I've created something on my own, which can build whole stacks of
packages and create apt repos. it also allows fine control on what is
in the base image, extra repos, etc.
The bad thing for me is: I've only got limited computing power and
and very limited in available archs (just x86 and some older arm).
So, having a CI that can build for all the debian supported archs and
allows using extra repos, tailored base images, work on git directly
(fully debianized branch) and publishes the repos to the outside world,
would be a really cool thing for me. IIRC that should also cover this
'scratch builds' usecase.
I admit, I haven't checked whether gitlab-ci can already do that.
--mtx
--
Enrico Weigelt, metux IT consult
Free software and Linux embedded engineering
i...@metux.net -- +49-151-27565287
Re: scratch buildds
On Sat, Jun 15, 2019 at 11:34:19PM +0200, Bernd Zeimetz wrote: > > Hi Chris, > > On 6/15/19 12:28 AM, Chris Lamb wrote: > > Adam Borowski wrote: > > > >> Thus, what would you guys say about a new distribution, "scratch"? It > >> would > >> be a kind of extra-experimental that doesn't put its build results anywhere > >> persistent. Throwing away built .debs would be ok, keeping just logs. > > > > Perhaps I'm missing something but would introducing more architectures > > to the salsa.debian.org continuous integrations runners not serve > > mostly the same purpose? The developer's workflow would simply be to > > push a commit and it would be built and tested automatically. > afaik the CI runners use k8s to schedule their work, so I think using > the default CI stuff from gitlab requires an architecture supported by > k8s. arm64 is supported and I know that some people cross-compiled k8s > for mips(el?), but I doubt its widely supported. > > Am I missing something? No, that's not the case. That is, it is *possible* to use kubernetes to do GitLab CI, but it is in no way a requirement. gitlab-runner has various executors to do stuff, and running something inside docker (with or without kubernetes) is just one of the many options. -- To the thief who stole my anti-depressants: I hope you're happy -- seen somewhere on the Internet on a photo of a billboard
Re: scratch buildds
On June 15, 2019 9:34:19 PM UTC, Bernd Zeimetz wrote: > >afaik the CI runners use k8s to schedule their work, so I think using >the default CI stuff from gitlab requires an architecture supported by >k8s. arm64 is supported and I know that some people cross-compiled k8s >for mips(el?), but I doubt its widely supported. > >Am I missing something? > Gitlab CI uses docker containers. At least that's been my experience with it. -Jim P.
Re: scratch buildds
Hi Chris, On 6/15/19 12:28 AM, Chris Lamb wrote: > Adam Borowski wrote: > >> Thus, what would you guys say about a new distribution, "scratch"? It would >> be a kind of extra-experimental that doesn't put its build results anywhere >> persistent. Throwing away built .debs would be ok, keeping just logs. > > Perhaps I'm missing something but would introducing more architectures > to the salsa.debian.org continuous integrations runners not serve > mostly the same purpose? The developer's workflow would simply be to > push a commit and it would be built and tested automatically. afaik the CI runners use k8s to schedule their work, so I think using the default CI stuff from gitlab requires an architecture supported by k8s. arm64 is supported and I know that some people cross-compiled k8s for mips(el?), but I doubt its widely supported. Am I missing something? Bernd -- Bernd ZeimetzDebian GNU/Linux Developer http://bzed.dehttp://www.debian.org GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F
Re: scratch buildds
On Sat, Jun 15, 2019 at 05:01:29PM +0200, Adam Borowski wrote: > Not every commit is worth testing, So only push when you want to test. GitLab CI tests every push, not every commit. > especially on bigger packages. I don't > want to cause unnecessary drain on already limited resources (crap > architectures have slow buildds). It's perfectly possible to create a GitLab CI configuration that says build:mipsel: tags: - mipsel when: manual # ... rest of config goes here where the "when: manual" means "create the job in the database, but don't start it unless someone logs on to the webinterface and clicks the 'run this job' button". That would alleviate that concern. -- To the thief who stole my anti-depressants: I hope you're happy -- seen somewhere on the Internet on a photo of a billboard
Re: scratch buildds
On Fri, Jun 14, 2019 at 11:28:47PM +0100, Chris Lamb wrote: > Adam Borowski wrote: > > > Thus, what would you guys say about a new distribution, "scratch"? It would > > be a kind of extra-experimental that doesn't put its build results anywhere > > persistent. Throwing away built .debs would be ok, keeping just logs. > > Perhaps I'm missing something but would introducing more architectures > to the salsa.debian.org continuous integrations runners not serve > mostly the same purpose? Yeah, with the "more architectures" being the key point. I haven't used salsa CI, but eg. Reproducible Builds have four archs: amd64, arm64, armhf, i386. I have machines that can run these without software emulation literally at hand's reach -- from each of three places I typically code at. That's not the case for any other arch, including a bunch of release ones. And I have only a sharply limited amount of damn to give for eg. mipsel. It's somehow still a release arch, thus as a DD I'm still supposed to at least try to port to it, but there's only so many tuits I, and other maintainers, have -- and for this reason I'd prefer porting to be convenient. > The developer's workflow would simply be to > push a commit and it would be built and tested automatically. Not every commit is worth testing, especially on bigger packages. I don't want to cause unnecessary drain on already limited resources (crap architectures have slow buildds). > (I would concede that this would essentially require adopting a salsa > and Git-based packaging scheme, however.) Or to develop elsewhere, and push to Salsa just to trigger CI. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ Don't be racist. White, amber or black, all beers should ⣾⠁⢠⠒⠀⣿⡁ be judged based solely on their merits. Heck, even if ⢿⡄⠘⠷⠚⠋ occasionally a cider applies for a beer's job, why not? ⠈⠳⣄ On the other hand, corpo lager is not a race.
Re: scratch buildds
On Sat, Jun 15, 2019 at 12:17:07AM +0200, Michael Biebl wrote: > It would be awesome if we had resources to run autopkgtests for such > scratch builds on a variety of archs as well. Hell yeah! -- ⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased ⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts. ⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got ⠈⠳⣄ agriculture, towns then cities. -- whitroth on /.
Re: scratch buildds
On Sat, Jun 15, 2019 at 11:14:37AM +0100, Colin Watson wrote: > Well, this is a false equivalence. I explicitly designed Ubuntu's > -proposed to be equivalent to unstable, rather than to a new thing that > Debian didn't have. > > (Albeit with some minor differences in detail: it's a partial suite > rather than a complete one, migration is much quicker, and it's more > firmly emphasised as something that's for machine consumption rather > than human. But the software that Ubuntu uses to promote from > -proposed to is the same as the software that Debian > uses to promote from unstable to testing.) I think that you're hiding a major difference here. The "for machine consumption" part is key imo. When the intended audience is a gating mechanism, unstable is the scratch space tha Adam asks for. When people have the expectation that unstable is being tested by humans and people expect unstable to be testable without too much frustration, we seek for other places to do pre-upload qa (e.g. salsa-ci). This is also relevant when doing distribution-wide QA. While lintian tends to be unaffected by breakage in other packages, build tests, piuparts, autopkgtest etc. Tend to get broken every now and then. The ftbfs bts tag somewhat helps with ignoring known breakage. Sorting through the many QA results is a time consuming task. Rejecting certain classes of failures earlier would help with reducing that load. In my experience, a relatively productive filter is to only consider those packages in unstable, that have some version in testing. Thanks to the autoremover. Helmut
Re: scratch buildds
On Sat, Jun 15, 2019 at 11:20:17AM +0200, Helmut Grohne wrote:
> On Fri, Jun 14, 2019 at 10:51:56PM +0200, Adam Borowski wrote:
> > Thus, what would you guys say about a new distribution, "scratch"? It would
> > be a kind of extra-experimental that doesn't put its build results anywhere
> > persistent. Throwing away built .debs would be ok, keeping just logs.
>
> I think this is inconvenient as well. As a developer, one has to wait
> and check the logs, then do the real upload. Wouldn't it be much better
> if a good build with no lintian errors, no autopkgtest failures, no
> piuparts failures, etc. would just move to unstable without a delay?
>
> Wait, this reminds me of something. There was this other distribution...
> Ubuntu! They have this ${dist}-proposed.
Well, this is a false equivalence. I explicitly designed Ubuntu's
-proposed to be equivalent to unstable, rather than to a new thing that
Debian didn't have.
(Albeit with some minor differences in detail: it's a partial suite
rather than a complete one, migration is much quicker, and it's more
firmly emphasised as something that's for machine consumption rather
than human. But the software that Ubuntu uses to promote from
-proposed to is the same as the software that Debian
uses to promote from unstable to testing.)
--
Colin Watson [cjwat...@debian.org]
Re: scratch buildds
On Fri, Jun 14, 2019 at 10:51:56PM +0200, Adam Borowski wrote:
> Thus, what would you guys say about a new distribution, "scratch"? It would
> be a kind of extra-experimental that doesn't put its build results anywhere
> persistent. Throwing away built .debs would be ok, keeping just logs.
I think this is inconvenient as well. As a developer, one has to wait
and check the logs, then do the real upload. Wouldn't it be much better
if a good build with no lintian errors, no autopkgtest failures, no
piuparts failures, etc. would just move to unstable without a delay?
Wait, this reminds me of something. There was this other distribution...
Ubuntu! They have this ${dist}-proposed. I think we've been discussing
this since at least 2013[1] (like bikesheds).
Unlike the ${dist}-proposed variant, the scratch distribution can be set
up entirely outside Debian. It only needs someone doing the work with no
involvement of DSA. Wait, this reminds me of something. Luca Falavigna
put up debomatic-${arch}.debian.net. And it has piuparts and lintian!
Looks like we already have scratch for years and nobody noticed.
I guess what we need now is something different: We've got lots of tools
and lots of tool diversity. Different people know different tools, some
of which solve the same tasks. A recent discussion on debhelper has
indicated that our tool diversity has a cost that not everyone is
willing to pay. As much was need diverse people in the project, we also
need to figure out how to reduce this tool diversity.
Helmut
[1] http://penta.debconf.org/dc13_schedule/events/1028.en.html
Re: scratch buildds
Adam Borowski wrote: > Thus, what would you guys say about a new distribution, "scratch"? It would > be a kind of extra-experimental that doesn't put its build results anywhere > persistent. Throwing away built .debs would be ok, keeping just logs. Perhaps I'm missing something but would introducing more architectures to the salsa.debian.org continuous integrations runners not serve mostly the same purpose? The developer's workflow would simply be to push a commit and it would be built and tested automatically. This approach would also appear to cover more use-cases, require less DSA attention ,and even allow non-DD/DMs to utilise the service too. (I would concede that this would essentially require adopting a salsa and Git-based packaging scheme, however.) Regards, -- ,''`. : :' : Chris Lamb `. `'` la...@debian.org chris-lamb.co.uk `-
Re: scratch buildds
On Fri, Jun 14, 2019 at 10:51:56PM +0200, Adam Borowski wrote: > Hi! > Fedora has an awesome feature for packagers: scratch builds. It would be > great if we could steal the idea. > > I find myself doing incremental uploads just to fix bugs that the previous > upload revealed on some weird arch. At home, I can reasonably test only > amd64 and arm64 -- especially if valgrind is involved, qemu-user is not up > to scratch. There are porterboxes but using them is inconvenient and > involved, especially when the task is "build on all archs, report failures". for host in $(porterhost1 porterhost2 porterhost3); do dcmd scp foo.dsc $host: ssh $host <
Re: scratch buildds
Am 14.06.19 um 22:51 schrieb Adam Borowski: > Hi! > Fedora has an awesome feature for packagers: scratch builds. It would be > great if we could steal the idea. > > I find myself doing incremental uploads just to fix bugs that the previous > upload revealed on some weird arch. At home, I can reasonably test only > amd64 and arm64 -- especially if valgrind is involved, qemu-user is not up > to scratch. There are porterboxes but using them is inconvenient and > involved, especially when the task is "build on all archs, report failures". > > Thus, what would you guys say about a new distribution, "scratch"? It would > be a kind of extra-experimental that doesn't put its build results anywhere > persistent. Throwing away built .debs would be ok, keeping just logs. > > Like any other upload currently, it would be restricted to DDs/DMs only -- > I'm told buildds have inadequate isolation to run untrusted builds, even > without taking into account container/VM escapes, etc. On IRC, Ansgar > requested keeping signed records as a precaution against hijacked DD > accounts; that idea sounds good to me. > > While every of us can (and in 99% cases does) test on amd64, it would be > nice to ease testing elsewhere as well. It would be awesome if we had resources to run autopkgtests for such scratch builds on a variety of archs as well. signature.asc Description: OpenPGP digital signature
scratch buildds
Hi! Fedora has an awesome feature for packagers: scratch builds. It would be great if we could steal the idea. I find myself doing incremental uploads just to fix bugs that the previous upload revealed on some weird arch. At home, I can reasonably test only amd64 and arm64 -- especially if valgrind is involved, qemu-user is not up to scratch. There are porterboxes but using them is inconvenient and involved, especially when the task is "build on all archs, report failures". Thus, what would you guys say about a new distribution, "scratch"? It would be a kind of extra-experimental that doesn't put its build results anywhere persistent. Throwing away built .debs would be ok, keeping just logs. Like any other upload currently, it would be restricted to DDs/DMs only -- I'm told buildds have inadequate isolation to run untrusted builds, even without taking into account container/VM escapes, etc. On IRC, Ansgar requested keeping signed records as a precaution against hijacked DD accounts; that idea sounds good to me. While every of us can (and in 99% cases does) test on amd64, it would be nice to ease testing elsewhere as well. Meow! -- ⢀⣴⠾⠻⢶⣦⠀ ⣾⠁⢠⠒⠀⣿⡁ Remember, the S in "IoT" stands for Security, while P stands ⢿⡄⠘⠷⠚⠋⠀ for Privacy. ⠈⠳⣄
