Re: [gentoo-dev] Suggestion: support the Dev team with system resources
08.11.2013 09:04, Johann Schmitz пишет: On 07.11.2013 21:18, Rich Freeman wrote: Seriously, though, I'd love to see these needs better supported. I think we need to start by defining what the needs actually are (less redundancy, more consistency, etc). Then we figure out how to best address them. It could be individuals donating VMs, or it might be Gentoo buying resources from any number of vendors, or it could be Gentoo going out and looking for donors. I agree with that. It's easier to decide what to do if we know what we need. A solution built by the infra team would be the best solution for the same reasons why it's better to put stuff on the devspace instead of private servers (availabilty; who can fix stuff, logins, etc). But if someone need resources and a box to play with I would happily provide an Xen instance. Just wondering: How is the AT for $minorarch done? Is it possible to run, say, mips on xen/whatever through some emulation layer or is real hardware a requirement for this archs? For the security concerns: I think these boxes should be used for testing only and not for development - every commit must be done from a box fully under the dev's control. Usually it is done via qemu, but as was said - it is damn slow. I use qemu for emulating ARM device for compilation, cause it's faster than compile on device(Raspberry Pi) itself mostly because of I/O. -- Best regards, Sergey Popov Gentoo developer Gentoo Desktop-effects project lead Gentoo Qt project lead Gentoo Proxy maintainers project lead signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
Johann Schmitz er...@gentoo.org writes: Is it possible to run, say, mips on xen/whatever through some emulation layer or is real hardware a requirement for this archs? Yes, via qemu. But very slow, nearly unusable even on a powerful mainstream amd64 server.
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
Dear Denis, Denis M. g...@politeia.in writes: Please review this, and if you agree that it'd be a good idea come with any suggestions to make it happen as well as with any other thoughts/sys-specs/instances we should be looking for. Thanks for the offering. Though not a member, AT teams might benefit from such a build farm. What are you suggesting practically, making a policy for everyone to donate VM to Gentoo, or developing a midware to do so? Cheers, Benda
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 12:53 PM, hero...@gentoo.org wrote: Dear Denis, Hi Benda, Denis M. g...@politeia.in writes: Please review this, and if you agree that it'd be a good idea come with any suggestions to make it happen as well as with any other thoughts/sys-specs/instances we should be looking for. Thanks for the offering. Though not a member, AT teams might benefit from such a build farm. Almost every Gentoo dev that does software testings of some sorts could benefit from these build farms (although I'd refrain from using that term ;) ..). What are you suggesting practically, making a policy for everyone to donate VM to Gentoo, or developing a midware to do so? My initial idea was to suggest this here (in the gentoo-dev@ ML) first and see what you guys think about the idea. If it gets accepted by majority, then a policy, rules, etc... should be gathered through your comments here. After that we could make a wiki page (as Ago suggested while we were talking about this in IRC) and spam the gentoo-user ML and see how many good people are there :-). Cheers, Benda Regards, Denis M. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On Thu, Nov 7, 2013 at 7:14 AM, Denis M. g...@politeia.in wrote: Almost every Gentoo dev that does software testings of some sorts could benefit from these build farms (although I'd refrain from using that term ;) ..). Don't let me put a damper on your plans as-is, but I'd be interested if developers who frequently perform these kinds of tasks post about what they're actually doing. Rather than just asking people to give random others ssh access to random boxes, it might make sense to streamline certain tasks. Imagine a tool that takes in a list of atoms and dumps a tarball of build logs in some standard layout. That could be easily distributed (assuming packages were reasonably independent), and tools like tatt might even be adapted. Not a reason to delay what you propose, just another opportunity.
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
Rackspace (where I work) currently has a developer discount program. I think we also host some open source stuff for various projects. Right now you can try to use http://developer.rackspace.com/ but if we want to make this more official I can ask around. Let me know if we want this as a more official thing (rackspace donating compute resources), no guarantees though :D. -- -- Matthew Thode (prometheanfire) signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 02:48 PM, Matthew Thode wrote: Rackspace (where I work) currently has a developer discount program. I think we also host some open source stuff for various projects. Right now you can try to use http://developer.rackspace.com/ but if we want to make this more official I can ask around. Let me know if we want this as a more official thing (rackspace donating compute resources), no guarantees though :D. To be honest, I would like Gentoo infra to come up with a solution sometime... Last time (a year ago) i asked them about this, they said they have a cluster/big box for this purpose but they just didn't have the time to deploy it properly or something. Not everyone can afford paid solutions when it comes to contributing to free software -- Regards, Markos Chandras
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 12:26 PM, Markos Chandras wrote: On 11/07/2013 02:48 PM, Matthew Thode wrote: Rackspace (where I work) currently has a developer discount program. I think we also host some open source stuff for various projects. Right now you can try to use http://developer.rackspace.com/ but if we want to make this more official I can ask around. Let me know if we want this as a more official thing (rackspace donating compute resources), no guarantees though :D. To be honest, I would like Gentoo infra to come up with a solution sometime... Last time (a year ago) i asked them about this, they said they have a cluster/big box for this purpose but they just didn't have the time to deploy it properly or something. Not everyone can afford paid solutions when it comes to contributing to free software iirc, we give $200 if infra for developer accounts for a couple of months. If a deal is struck it would likely be more and forever or something. -- -- Matthew Thode (prometheanfire) signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 08:59 PM, Matthew Thode wrote: On 11/07/2013 12:26 PM, Markos Chandras wrote: On 11/07/2013 02:48 PM, Matthew Thode wrote: Rackspace (where I work) currently has a developer discount program. I think we also host some open source stuff for various projects. Right now you can try to use http://developer.rackspace.com/ but if we want to make this more official I can ask around. Let me know if we want this as a more official thing (rackspace donating compute resources), no guarantees though :D. To be honest, I would like Gentoo infra to come up with a solution sometime... Last time (a year ago) i asked them about this, they said they have a cluster/big box for this purpose but they just didn't have the time to deploy it properly or something. Not everyone can afford paid solutions when it comes to contributing to free software iirc, we give $200 if infra for developer accounts for a couple of months. If a deal is struck it would likely be more and forever or something. I've been running my VM for Ago for 13 months now (started on september 2012), where are my $200? ;-) Regards, Denis M. (Phr33d0m) signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On Thu, Nov 7, 2013 at 3:08 PM, Denis M. g...@politeia.in wrote: On 11/07/2013 08:59 PM, Matthew Thode wrote: iirc, we give $200 if infra for developer accounts for a couple of months. If a deal is struck it would likely be more and forever or something. I've been running my VM for Ago for 13 months now (started on september 2012), where are my $200? ;-) Can't argue with that. :) Seriously, though, I'd love to see these needs better supported. I think we need to start by defining what the needs actually are (less redundancy, more consistency, etc). Then we figure out how to best address them. It could be individuals donating VMs, or it might be Gentoo buying resources from any number of vendors, or it could be Gentoo going out and looking for donors. I suspect that if we went out with something specific in mind we might be able to find a sponsor - but it is always best to have some idea just what we're going to be using any donations for (this will be our stage3 builder which cranks out a new stage3 every 20 minutes and reports build failures to double as a tinderbox, etc). Rich
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 09:18 PM, Rich Freeman wrote: On Thu, Nov 7, 2013 at 3:08 PM, Denis M. g...@politeia.in wrote: On 11/07/2013 08:59 PM, Matthew Thode wrote: iirc, we give $200 if infra for developer accounts for a couple of months. If a deal is struck it would likely be more and forever or something. I've been running my VM for Ago for 13 months now (started on september 2012), where are my $200? ;-) Can't argue with that. :) Seriously, though, I'd love to see these needs better supported. I think we need to start by defining what the needs actually are (less redundancy, more consistency, etc). Then we figure out how to best address them. It could be individuals donating VMs, or it might be Gentoo buying resources from any number of vendors, or it could be Gentoo going out and looking for donors. I suspect that if we went out with something specific in mind we might be able to find a sponsor - but it is always best to have some idea just what we're going to be using any donations for (this will be our stage3 builder which cranks out a new stage3 every 20 minutes and reports build failures to double as a tinderbox, etc). Rich Currently Diego's tinderbox does something like that AFAIK. Compiles things and (almost?) automatically submits bugs against the packages with the relevant logs, etc... The initial idea behind my suggestion was that the devs would have the enough system resources to address these bugs (and the ones reported from the users, of course). An example here could be the following: finding/confirming a compilation bug for a package with ~10 USE flags could take tatt quite some compilations depending on the USE flag's combinations (this is actually what arch testers do in order to stabilize/keyword a package). Another example would be, as I mentioned in my previous mails to this thread - a new glibc version comes out and (as you know) quite some packages fail to compile against it. Having the resources, it would be possible to track these packages faster instead of relying on random users/testers to report them to bugs.g.o. And a last one would be testing new KDE/GNOME/whatever-meta-with-huge-number-of-packages. As an AT member myself I could only give examples on how using such system of donating/providing instances would be a benefit. For a comprehensive list of the tasks (for consistency as you said), I'd wait for actual devs to enumerate their needs. I doubt this will go as further as Gentoo actually *buying* resources. The reason is obvious - things have been going fine till now, why throw monnies for something as 'unnecessary' (which is why I haven't received a penny for it, hehehe), that's why I came with the donorship-of-instances version. I believe the 'going out looking for donors' part you said is basically what I'm suggesting here, although I believe you meant donors = huge companies providing clusters, and I doubt that'll happen. From my observation, you can get a lot of work done on a simple 2GB-ram-4-cores VirtualBox VM. Not to talk that lots of people nowadays have these resources to spare. That's why getting actual people (and not companies or whatever) to donate their system resources is easier to get/reach. Regards, Denis M. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 07/11/13 09:20 AM, Rich Freeman wrote: On Thu, Nov 7, 2013 at 7:14 AM, Denis M. g...@politeia.in wrote: Almost every Gentoo dev that does software testings of some sorts could benefit from these build farms (although I'd refrain from using that term ;) ..). Don't let me put a damper on your plans as-is, but I'd be interested if developers who frequently perform these kinds of tasks post about what they're actually doing. Rather than just asking people to give random others ssh access to random boxes, it might make sense to streamline certain tasks. Imagine a tool that takes in a list of atoms and dumps a tarball of build logs in some standard layout. That could be easily distributed (assuming packages were reasonably independent), and tools like tatt might even be adapted. Not a reason to delay what you propose, just another opportunity. I guess nobody wants to try and setup a VM-image-based heterogeneous grid system, huh? :) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlJ8AcIACgkQ2ugaI38ACPCqHwEAulNSjBvU4WsLu91zChM8esBf M7FWlAdM++LUsfZ0y/cA/3oZp4+7mjeWbJdUlNxtAGBDYYxD9WfNzpitwX0IFWnN =q61v -END PGP SIGNATURE-
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 03:07 PM, Denis M. wrote: On 11/07/2013 09:18 PM, Rich Freeman wrote: On Thu, Nov 7, 2013 at 3:08 PM, Denis M. g...@politeia.in wrote: On 11/07/2013 08:59 PM, Matthew Thode wrote: iirc, we give $200 if infra for developer accounts for a couple of months. If a deal is struck it would likely be more and forever or something. I've been running my VM for Ago for 13 months now (started on september 2012), where are my $200? ;-) Can't argue with that. :) Seriously, though, I'd love to see these needs better supported. I think we need to start by defining what the needs actually are (less redundancy, more consistency, etc). Then we figure out how to best address them. It could be individuals donating VMs, or it might be Gentoo buying resources from any number of vendors, or it could be Gentoo going out and looking for donors. I suspect that if we went out with something specific in mind we might be able to find a sponsor - but it is always best to have some idea just what we're going to be using any donations for (this will be our stage3 builder which cranks out a new stage3 every 20 minutes and reports build failures to double as a tinderbox, etc). Rich Currently Diego's tinderbox does something like that AFAIK. Compiles things and (almost?) automatically submits bugs against the packages with the relevant logs, etc... The initial idea behind my suggestion was that the devs would have the enough system resources to address these bugs (and the ones reported from the users, of course). An example here could be the following: finding/confirming a compilation bug for a package with ~10 USE flags could take tatt quite some compilations depending on the USE flag's combinations (this is actually what arch testers do in order to stabilize/keyword a package). Another example would be, as I mentioned in my previous mails to this thread - a new glibc version comes out and (as you know) quite some packages fail to compile against it. Having the resources, it would be possible to track these packages faster instead of relying on random users/testers to report them to bugs.g.o. And a last one would be testing new KDE/GNOME/whatever-meta-with-huge-number-of-packages. As an AT member myself I could only give examples on how using such system of donating/providing instances would be a benefit. For a comprehensive list of the tasks (for consistency as you said), I'd wait for actual devs to enumerate their needs. I doubt this will go as further as Gentoo actually *buying* resources. The reason is obvious - things have been going fine till now, why throw monnies for something as 'unnecessary' (which is why I haven't received a penny for it, hehehe), that's why I came with the donorship-of-instances version. I believe the 'going out looking for donors' part you said is basically what I'm suggesting here, although I believe you meant donors = huge companies providing clusters, and I doubt that'll happen. From my observation, you can get a lot of work done on a simple 2GB-ram-4-cores VirtualBox VM. Not to talk that lots of people nowadays have these resources to spare. That's why getting actual people (and not companies or whatever) to donate their system resources is easier to get/reach. Regards, Denis M. I may also have a small openstack cluster I can let people use soonish. Working on a backlog of issues now. -- -- Matthew Thode (prometheanfire) signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07.11.2013 21:18, Rich Freeman wrote: Seriously, though, I'd love to see these needs better supported. I think we need to start by defining what the needs actually are (less redundancy, more consistency, etc). Then we figure out how to best address them. It could be individuals donating VMs, or it might be Gentoo buying resources from any number of vendors, or it could be Gentoo going out and looking for donors. I agree with that. It's easier to decide what to do if we know what we need. A solution built by the infra team would be the best solution for the same reasons why it's better to put stuff on the devspace instead of private servers (availabilty; who can fix stuff, logins, etc). But if someone need resources and a box to play with I would happily provide an Xen instance. Just wondering: How is the AT for $minorarch done? Is it possible to run, say, mips on xen/whatever through some emulation layer or is real hardware a requirement for this archs? For the security concerns: I think these boxes should be used for testing only and not for development - every commit must be done from a box fully under the dev's control. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSfHDMAAoJEKCEBkJ3xQHt2NgH/RxKb8nQDTnpjmTjkiJs/i04 JC36jxOj/ZMSSmyayssw/lpIHVB0z3V+nypLwDZnoTR5AfqQZ2O63G2OUSQwl0MN SCHYNvrQrqxPeRmQ8SBP8VMiDK6vClgRSSnJaRAKKI+ZzpDVf5BjljRr4YeakV/t iEvVpWeFt+gRDZBdFL2mInkbJ+3QBuPU08PS2p2mdrfZ3/b046eqZBQcmjnIk2/r rfVkaQ69IzS90tvv55AM3jjGIFxa/Fh5eIw7CC/VCyhiqH2egRfTTaCfdFz4VWTs 2IWNuwK3K9hxiCxzsH+IvLtqIvNYVXHdqy/6JfcIfGdlEI7/rdk2/I8VpWaOKy0= =36Sm -END PGP SIGNATURE-
[gentoo-dev] Suggestion: support the Dev team with system resources
Hello gentoo-dev@, Starting with a little intro, I'm currently providing a Gentoo VM to a gentoo dev (Agostino Sarubbo (ago)) for the purpose of testing/stabilizing/keywording packages, which is part of his task as a developer and being part of the AT team. I've been running the VM for him for a couple of months now and AFAIK he's been giving it a great use ;-). The main idea here is to allow Gentoo contributors and members (not necessary) of the Gentoo community, to be able to support the developer team providing their spare system resources, by, for example, running a Virtual Machine (or any sort of xen, kvm, virtualbox, vmware, whatever...) instance where the devs can run tasks they'd normally wouldn't be able to run with their systems, because: * They're doing some other tests at the moment * They're on ~arch and need stable * They're not on the architecture needed for that testing * Their system is not 'powerful' enough * etc... The purpose of doing this is that the developers that have the time and dedication would be able to run a couple of different tests concurrently, on different 'instances' provided by the community. That will greatly, IMHO, improve the team's performance and not only in the AT field. The instances provided wouldn't forcefully need to meet any specific minimum requirements (this would be decided once (and if) this gets accepted), but a dual core system with 512MB ram would be somewhat an acceptable instance for the bigger arches (x86 amd64), and maybe lower specs for the other arches[1]. As an example here, I'm giving Ago a VirtualBox VM with 2GB ram and 4 virtual CPUs. Also, for the contributors there shouldn't be any minimum uptime to meet, they'll run the instances the time they use their systems, and if they leave them idle all day/night that would just be better, although they should be able to specify to the team normally the hours their systems would be usable by the devs. There should be a list of users that are able to share their resources and each dev(s) would be given a certain number of instances depending on their needs and such. I know that you might think that doing this will lower the contributor's desktop experience (as VMs tend to be somewhat heavy while compiling). The usage of the AUTOGROUP kernel scheduler and cgroups tends to make the desktop very much usable under high CPU pressure. Please review this, and if you agree that it'd be a good idea come with any suggestions to make it happen as well as with any other thoughts/sys-specs/instances we should be looking for. If you don't think this is a good idea or that it won't profit the Gentoo dev team, please tell me why. Regards, Denis M. (Phr33d0m) PS: This is a re-send as I firstly sent it without subscribing to the ML. So sorry if you receive it 2 times. [1] I apologize if this statement is wrong, it's based of my 0 knowledge on the other arches and the resources they need. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
Am Donnerstag, 7. November 2013, 00:18:19 schrieb Denis M.: Hello gentoo-dev@, Starting with a little intro, I'm currently providing a Gentoo VM to a gentoo dev (Agostino Sarubbo (ago)) for the purpose of testing/stabilizing/keywording packages, which is part of his task as a developer and being part of the AT team. I've been running the VM for him for a couple of months now and AFAIK he's been giving it a great use ;-). The main idea here is to allow Gentoo contributors and members (not necessary) of the Gentoo community, to be able to support the developer team providing their spare system resources, by, for example, running a Virtual Machine (or any sort of xen, kvm, virtualbox, vmware, whatever...) instance where the devs can run tasks they'd normally wouldn't be able to run with their systems, because: ... I appreciate the idea, but security-wise it's pretty dangerous - given that you as a Gentoo dev are doing sensitive work that may affect many people on a machine not controlled by you yourself nor Gentoo Infra. Call me paranoid, but please no. And in absolutely no case one should commit to the tree from such a machine, even with stuff like agent forwarding. -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Suggestion: support the Dev team with system resources
On 11/07/2013 12:37 AM, Andreas K. Huettel wrote: Am Donnerstag, 7. November 2013, 00:18:19 schrieb Denis M.: Hello gentoo-dev@, Starting with a little intro, I'm currently providing a Gentoo VM to a gentoo dev (Agostino Sarubbo (ago)) for the purpose of testing/stabilizing/keywording packages, which is part of his task as a developer and being part of the AT team. I've been running the VM for him for a couple of months now and AFAIK he's been giving it a great use ;-). The main idea here is to allow Gentoo contributors and members (not necessary) of the Gentoo community, to be able to support the developer team providing their spare system resources, by, for example, running a Virtual Machine (or any sort of xen, kvm, virtualbox, vmware, whatever...) instance where the devs can run tasks they'd normally wouldn't be able to run with their systems, because: ... I appreciate the idea, but security-wise it's pretty dangerous - given that you as a Gentoo dev are doing sensitive work that may affect many people on a machine not controlled by you yourself nor Gentoo Infra. I completely agree with this, but it's not entirely true. Why? I'll give the example of the AT team: 1. You sync the tree before you start your work (that way you verify the tree is clean). 2. Then you start testing the packages or bugs you're after, which in matter of security is meaningless because testing packages is usually just compiling and running to see if it works as expected. 2.1. Apply random patches to fix if there's an issue. 2.2. goto 2. 3. etc... I see no issue in this in matter of security. Another example would be devs testing packages under development (internal usage in gentoo), for example how new versions of openrc/systemd/glibc/whatever can affect X. I do understand your concern, although I wouldn't call you paranoid as it's just normal to not trust a system that's not completely under your control, but as I said, you don't really... 'care' about it/that. Call me paranoid, but please no. And in absolutely no case one should commit to the tree from such a machine, even with stuff like agent forwarding. Of course! Commiting or any other form of direct communication with the gentoo infra. (either commit to tree or `git push`-ing to any of the other gentoo repos) would be highly discouraged, and I didn't, in any moment, think someone would think of doing that :P. The idea behind this is using the provided instance only and exclusively for testing something you'd normally can't do on your system. Regards, Denis M. signature.asc Description: OpenPGP digital signature
