Re: Debian services and Debian infrastructure
Hi, On Sat, 04 Jan 2014, Lucas Nussbaum wrote: I've given some thought to this myself, and came up with the following ideas. Some of them are probably really bad ideas, but let's try to brainstorm a bit: I don't find them bad. At least from the POV of view of a DD and of a service maintainer, they seem to go in the right direction. 1. to improve communication between DSA and service maintainers + have an archived, public list where (prospective) service maintainers can engage with DSA about stuff they are planning/thinking about. (Maybe recycle debian-admin@l.d.o, or use debian-services-admin@l.d.o?) debian-services-admin@l.d.o seems to be rather appropriate for this (it's unlikely that this kind of discussion would generate so much mail that the list could no longer serve its current purpose). + have DSA provide collective answers more often, rather than a set of individual answers. It's often not clear if something is a final decision, or the opinion of a DSA member. + redirect some discussions from IRC to a mailing list, esp. when things look like policy decisions (on service design, etc) It would also be helpful to have a document explaining their usual requirements. It would pro-actively direct the service maintainers towards choices that are acceptable to DSA. Things like: - use PostgreSQL if you need a DB - keep in mind that DSA likes to ensure high-availability, so make it easy to run a replicate of your service - avoid dependencies not in stable or stable-backports - avoid X, Y, Z for security reasons - etc. 2. to improve the tracking of services + request wiki pages from maintainers that detail who are the contact points, what the service does, what are its requirements. State a service level agreement (informative of expectations, not punitive, of course) + have service maintainers create similar pages for services in development, to provide some kind of incubation process during which DSA can provide feedback. This would be good to have but (someone|DSA) would have to prod service maintainers to get this started, otherwise it will never happen. This requirement could also be mentioned in the document I suggested earlier. 3. to provide a place to experiment with new services + create a Debian cloud with virtual machines to develop new services (maybe providing manually-created VMs would be enough -- I'm not sure we need a complex infra such as OpenStack). I think we're already close to this at least in terms of creating virtual machines for new services. I got one created rather easily and several others were added recently. That said a Debian cloud for short-lived Debian tasks is still an interesting ideas, possibly in relation with PPA and having the possibility to rebuild packages against a PPA. Q1. How much should we push for Debian services (services useful to Debian) to be hosted on Debian infrastructure? Fragmentation in terms of hosting is harmful, and we should all try to get our services hosted on Debian infrastructure, because that's the easiest way to ensure that the maintainance of such services can be shared or transferred between developers. Now, for some services, it seems that doing this has become so hard that it harmed the service itself, and badly demotivated the service maintainers. I don't think that we should allow that to happen. We should stay open to other hosting solutions, when this is necessary. Having stuff hosted on DSA-managed hardware seems like something important to me. That said we can't force all services to be ready to meet DSA's requirements from day 1. So my suggestion would be have DSA allocate VMs to host services in incubation and keep those services in the .debian.net umbrella until they are ready. There's always going to be exceptions (stuff needing so much resource that it's unrealistic to allocate them out of the current hardware pool) but that's OK. Those can be dealt on a case by case basis and are likely to benefit from early interaction with DSA. Q2. Should we seek other hosting opportunities to ease Debian services development and hosting? Should I authorize the use of Debian money to fund infrastructure not managed by DSA, in the case of a useful service that DSA has been unable to accept in the infrastructure it manages? For services development (something for which we don't have a good answer yet), I think that we should explore getting specific sponsored hosting. As you know, Amazon provides Debian with free credits on the AWS Cloud. Those credits are normally used for archive rebuilds, but have already been used by a GSOC project (PTS rewrite), and more recently, to start work on an archive-wide autopkgtest setup. A few more virtual machines could be accomodated, and I will investigate if we could extend that even more. Also, codesearch.d.n is hosted on Rackspace's
Re: Debian services and Debian infrastructure
On Sun, Jan 05, 2014 at 12:05:18PM +0100, Raphael Hertzog wrote: 1. to improve communication between DSA and service maintainers + have an archived, public list where (prospective) service maintainers can engage with DSA about stuff they are planning/thinking about. (Maybe recycle debian-admin@l.d.o, or use debian-services-admin@l.d.o?) debian-services-admin@l.d.o seems to be rather appropriate for this (it's unlikely that this kind of discussion would generate so much mail that the list could no longer serve its current purpose). debian-services-admin@l.d.o seems to me, too, to be the perfect place for it. It would also be helpful to have a document explaining their usual requirements. It would pro-actively direct the service maintainers towards choices that are acceptable to DSA. Things like: - use PostgreSQL if you need a DB - keep in mind that DSA likes to ensure high-availability, so make it easy to run a replicate of your service - avoid dependencies not in stable or stable-backports - avoid X, Y, Z for security reasons - etc. Also agreed. It sounds to me like a specifications of the production environment where things will be deployed, and it's something that as a developer I should see sooner rather than later. 2. to improve the tracking of services + request wiki pages from maintainers that detail who are the contact points, what the service does, what are its requirements. State a service level agreement (informative of expectations, not punitive, of course) + have service maintainers create similar pages for services in development, to provide some kind of incubation process during which DSA can provide feedback. This would be good to have but (someone|DSA) would have to prod service maintainers to get this started, otherwise it will never happen. This requirement could also be mentioned in the document I suggested earlier. I think we already have something very close to this description here: https://wiki.debian.org/Services Ciao, Enrico -- GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini enr...@enricozini.org signature.asc Description: Digital signature
Re: Debian services and Debian infrastructure
On 13446 March 1977, Lucas Nussbaum wrote: Q1. How much should we push for Debian services (services useful to Debian) to be hosted on Debian infrastructure? A lot. Should I authorize the use of Debian money to fund infrastructure not managed by DSA, in the case of a useful service that DSA has been unable to accept in the infrastructure it manages? Like wasting money on essentially dead systems, like m68k? However, there has been several episodes, involving the development of new services or their move to Debian infrastructure, that generated a lot of frustration and demotivation on both sides (DSA and service developers). As examples, one can think of codesearch, or the fedmsg GSOC project. From what I followed on some service, a good part of the frustration came from real unrealistic requirements that just got thrown in front of DSA. One thing i remember is Oh, we just need fast storage, a recent SSD is enough [...] its only a hundred or so dollars - which is an easy thing to say talking about desktop machines where you put one. But DSA operates on a different level and suddenly putting SSD storage came out multiple thousand dollars. Not surprising that it wasnt taken all that good a suggestion. And another can be summarized to here, take this, no, your ideas aren't considered, all the rest of the world eats this too, so you have to eat it also. Makes people love to help you. There are also services that have been developed outside of Debian's infrastructure, with AFAIK no plans to move them to Debian infrastructure. The recurring pattern seems to be that, when DSA is approached to move the service to Debian infrastructure, their evaluation of the service's design results in changes requests or constraints that the service maintainers consider too hard to satisfy. And that is a problem. Not directly with DSA, not directly with the service maintainers, but with the structure of having development of services done outside the infrastructure it later on should run on and then blindly assuming that the infrastructure can just take it. That does not work. I'm worried that this situation is harmful for the project. Yes, seperate development with different requirements than the production environment is harmful. It's a lesson that every larger company with an IT Department goes through at some point - and it's the same for our project. You need a development environment that is build and handled the same way as the production environment. Sure you need to evolve that, it may not stand still, with new requirements coming along, but you need to do that in close work with the people maintaining the production one, to ensure that the way thinks evolve is actually something that can be mirrored in the area it is intended to be run later. (Alternatively you can have a free-for-all run development and then get a third env, in which you then try to get to run the new release in a (newer version of the) production env). Similarly to how we increased the amount of collaborative packages maintainance over the years, we should improve on collaborative service maintainance, and we should move away from the myriad of unofficial services maintained by a sole DD, and hosted on this DD's personal machine. Oh yes, I don't think anyone will disagree with Services for/around Debian should be run on official hosted resources. 1. to improve communication between DSA and service maintainers + have an archived, public list where (prospective) service maintainers can engage with DSA about stuff they are planning/thinking about. (Maybe recycle debian-admin@l.d.o, or use debian-services-admin@l.d.o?) + Have service maintainers accept that other people (eg DSA) have knowledge too. And accept their input. + have DSA provide collective answers more often, rather than a set of individual answers. It's often not clear if something is a final decision, or the opinion of a DSA member. If it gets a requirement that DSA has to give collective answers and otherwise should refrain from commenting, then its bad. Also, an individual DSA answering and giving his opinion shouldn't be a problem - it only would if the team itself is splitted and you get 5 opinions from 3 people. 2. to improve the tracking of services + request wiki pages from maintainers that detail who are the contact points, what the service does, what are its requirements. State a service level agreement (informative of expectations, not punitive, of course) + have service maintainers create similar pages for services in development, to provide some kind of incubation process during which DSA can provide feedback. + have service maintainers engage with DSA *early*, as early as remotely possible, listen to them and work with them to get useful (for both sides) compromises to make the service a good guest on Debian
Re: Debian services and Debian infrastructure
On Sun, Jan 05, 2014 at 02:28:59PM +0100, Joerg Jaspert wrote: Q3. What should be our definition of official services? Even if this is highly preferable, I don't think that official services (.d.o services) should necessarily be running on Debian hardware managed by DSA, provided that: - the service is clearly useful and used - the service has a sustainable maintainance model (active team + instructions on how to contribute, run a local copy, etc. + DFSG-free) - the service's design does not raise security or scalability concerns I disagree on that, official services should run under project overview. So far that the project can take it over and move it, should all of the team go away. Active team today doesn't mean they are there tomorrow to properly hand it over. Having the project itself have access (via DSA at least) ensures it can continue. I very much agree with Joerg on this. A team like DSA offers to the Debian Project two key features. One, which is the most visible one, is the technical output and continued service of a team of talented sysadmins. The other is the governance guarantee that we, as a project, can always take over the maintenance of a service whose current maintainers go MIA or become unwilling to work with the rest of the project for any reason. As Joerg points out. I understand that we can come up with a list of requirements that diminish the risk. This is, I believe, what Lucas was trying to do with the requirements quoted above. But I think it is much better to centralize those requirements on a single, DPL-delegated team, than trying to replicate that to many different, de facto self-proclaimed teams. Cheers. -- Stefano Zacchiroli . . . . . . . z...@upsilon.cc . . . . o . . . o . o Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o Former Debian Project Leader . . @zack on identi.ca . . o o o . . . o . « the first rule of tautology club is the first rule of tautology club » signature.asc Description: Digital signature
Re: Debian services and Debian infrastructure
We've implemented OpenNebula here at Brainfood and after having had some experience with it I would say that having that kind of infrastructure simplifies management rather than making it more complicated. Having a bunch of hand baked KVMs running without coordinated migration or shared storage is a lot more fragile, confusing and slow to work with. For instance, if we need to service a machine we can tell OpenNebula to flush that host and it will automatically balance all the virtual machines running on that machine to other hosts. That's a big convenience. I'm sure OpenStack has similar capabilities. We evaluated it along with a number of other infrastructures and for us OpenNebula was just the easiest to get going from the stock Debian packages. We also have found it to be very hackable. A lot of its functionality actually lives in shell scripts rather than some DSL or Ruby code. - Lucas Nussbaum lea...@debian.org wrote: 3. to provide a place to experiment with new services + create a Debian cloud with virtual machines to develop new services (maybe providing manually-created VMs would be enough -- I'm not sure we need a complex infra such as OpenStack). -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/27143708.9661388951071865.javamail.r...@newmail.brainfood.com
Re: Debian services and Debian infrastructure
On 13447 March 1977, Ean Schuessler wrote: 3. to provide a place to experiment with new services + create a Debian cloud with virtual machines to develop new services (maybe providing manually-created VMs would be enough -- I'm not sure we need a complex infra such as OpenStack). We've implemented OpenNebula here at Brainfood and after having had some [opennebula usage description] Note that DSA is not manually managing each little VM. They do use Ganeti. I'm not up comparing that to any cloud stuff, but it might do enough for what is suggested to have. -- bye, Joerg Some NM: Essential: Yes -- useful for a message when you do apt-get remove bash: -- To UNSUBSCRIBE, email to debian-project-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87vbxym9lk@gkar.ganneff.de
