Re: leaks in our only-signed-software fortress
On Mon, 2012-02-20 at 19:50 -0500, Michael Gilbert wrote: > But anyway, I think to get anywhere you'll need to help get Debian > policy 2.2.1 clarified for these kind of conditions. Then you'll be > able to submit bugs with appropriate RC severity so they'll have to be > handled. Phew,.. changing the policy is a terrible quest ;) And honestly, I don't think that all that is necessary can be coded in a policy. Especially as much is a best effort thing... like getting a trust path to upstream, or if this is not possible, download the sources from multiple different computers, etc. And we have many cases, where maintainers would really have to patch software, to prevent it from possibly doing nasty things (take all the packages with AppStore like stuff as an example, Mozilla Add-Ons, GNOME Shell Extensions, etc.) Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: leaks in our only-signed-software fortress
On Tue, 2012-02-21 at 16:59 -0600, Gunnar Wolf wrote: > Sadly, I think this is more propaganda and wishful thinking than > reality. And if I'm going to badmouth somebody, I'll badmouth myself. I guess you're right, that for large software it's difficult to impossible for the maintainer to really follow up all the code changes (basically doing an audit)... but still, what was said in this thread would help,... a) by carefully checking hashsums you at least prevent that single Debian users are attacked b) maintainers should still try to get a direct-as-possible trust-path to upstream c) we have many maintainers who take part in upstream, too, just take Mike as an example who has commit rights to Mozilla since some time, IIRC. Guess for such guys it might be possible to roughly track what has changed. d) for very small programs, especially when they don't change a lot and get just bugfixes, I can imagine, that some maintainers have a look at what has changed. Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: leaks in our only-signed-software fortress
I demand that Toni Mueller may or may not have written... > On 02/18/2012 11:48 AM, Thomas Koch wrote: >> What about a debhelper script that receives an URL (or set of mirror >> URLs) and a SHA1 and does the download and check? > If you're going this way, try to peek at the *BSD's ports systems, > specifically their 'distinfo' files. SHA1 is not enough, imho. For *xine* releases, I use MD5, SHA1 and SHA256. The hashes are then signed using gpg. That's mainly for others, though; I Don't Need to check them when doing packaging work for Debian. -- | _ | Darren Salt, using Debian GNU/Linux (and Android) | ( ) | | X | ASCII Ribbon campaign against HTML e-mail | / \ | http://www.asciiribbon.org/ A clean, neat, desk is a sign of a very sick mind. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/526d2783f4%li...@youmustbejoking.demon.co.uk
Re: leaks in our only-signed-software fortress
On Tue, 21 Feb 2012, Gunnar Wolf wrote: > Henrique de Moraes Holschuh dijo [Sat, Feb 18, 2012 at 10:46:50AM -0200]: > > Good packaging developers go to great lengths to be sure they are not > > going to distribute anything trojaned. This takes a lot of work, and > > often requires very goot working relationship with upstream to the point > > of getting upstream to change his processes. This does include tracking > > deviations from VCS to upstream releases, going over upstream changes > > when possible, and using crypto properly to verify authenticity of > > upstream commits and tarballs (when available. When it is not > > available, educating upstream about it is required). > > Sadly, I think this is more propaganda and wishful thinking than > reality. And if I'm going to badmouth somebody, I'll badmouth myself. Well, for all the stuff I've been downstream of, I think I've managed to do it for everything but hplip. It really means you have to track upstream VCS and look at the changes very often, because when they pile up... for hplip, I had to do it by skimming the changes and looking for obvious crap, as the volume was too big and it came as a code dump instead of incremental changes. And even just skimming the changes took a few hours, and I had to just plain ignore any changes to the PPD files or I'd go insane. Closed development and code dumps are nasty. Tracking deviations is a matter of instrumenting things properly, and it adds barely an hour per release after you've got things set up. Basically, it requires a good VCS, tracking upstream VCS, and tracking upstream tarball/whatever releases, and using the VCS diff :p When there is no upstream VCS, there are no deviations to track, and likely the most you can do is to pester upstream to do proper release signatures. It can get worse. The worst upstream I have about it so far is Intel's processor microcode people. Unsigned releases, absolutely impossible to contact upstream except indirectly by asking favours of some Intel employees that can be found on LKML, what looks from the outside to be an absolute nutjob of a microcode release management... The best I could do there was to expose the weirdness as a commented changelog of the timeline of each microcode[1], track deviations in each microcode (hash every opaque blob inside the metadata container and track it across every public microcode release since 2008 -- no deviations found to date), and add some guard time between upstream releases and Debian releases. [1] The package has the full changelog, but the relevant portions for each new release are also in the Debian changelog: http://packages.debian.org/changelogs/pool/non-free/i/intel-microcode/current/changelog > So, either I am among Debian's biggest liabilities, or your paragraph > reflects what we want others to think about us. My packages tend not > to break, and I think they meet Debian's standards, but they are far > from audited by me. You should not limit yourself to just that paragraph when reading what I wrote. It is far less about auditing, and far more about doing what is possible to try to shield the users from trojans added by a third-party to the release distribution points and public VCS (and not by a rogue upstream -- that one can be avoided only if you audit everything, and that's often downright impossible). Only you can judge whether you could/should/need to do better. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120222135808.ga21...@khazad-dum.debian.net
Re: leaks in our only-signed-software fortress
Henrique de Moraes Holschuh dijo [Sat, Feb 18, 2012 at 10:46:50AM -0200]: > Good packaging developers go to great lengths to be sure they are not > going to distribute anything trojaned. This takes a lot of work, and > often requires very goot working relationship with upstream to the point > of getting upstream to change his processes. This does include tracking > deviations from VCS to upstream releases, going over upstream changes > when possible, and using crypto properly to verify authenticity of > upstream commits and tarballs (when available. When it is not > available, educating upstream about it is required). Sadly, I think this is more propaganda and wishful thinking than reality. And if I'm going to badmouth somebody, I'll badmouth myself. Depending on the project this is about, I'll check different things. Some of my packages are quite big, and to be honest, more complex than what I can understand (so it could be argued I was irresponsible for packaging them to begin with). For those, I usually look at upstream's changelog or announcement, and try to match them with the open bugs in the BTS. If the upstream announcement includes checksums, I'll (often, at least) verify the tarballs I get. But I don't check the bits of diff between two revisions, surely not for large changes. In the case of smaller packages (most of what I maintain are libraries I use for my systems), I often check if they are still offer a coherent API, by trying my own stuff on them before uploading. Whenever the code includes test suites, I include them. However, I do _not_ audit the code itself. So, either I am among Debian's biggest liabilities, or your paragraph reflects what we want others to think about us. My packages tend not to break, and I think they meet Debian's standards, but they are far from audited by me. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120221225931.gc1...@gwolf.org
Re: leaks in our only-signed-software fortress
On 02/18/2012 11:48 AM, Thomas Koch wrote: > What about a debhelper script that receives an URL (or set of mirror URLs) > and > a SHA1 and does the download and check? If you're going this way, try to peek at the *BSD's ports systems, specifically their 'distinfo' files. SHA1 is not enough, imho. -- Kind regards, --Toni++ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f440fb6.5020...@debian.org
Re: leaks in our only-signed-software fortress
On Fri, Feb 17, 2012 at 11:09 PM, Christoph Anton Mitterer wrote: > For many of those I've reported bugs (and I'm sure I didn't found a lot of > them, and I'm further sure > that new cases were introduced). > Some where closed, some where just ignored or denied. Fortunately, this is rather uncommon. foo2jzs with its getweb printer firmware fetching script is the only one right now that I am aware of. I started a bug/conversation on that a few years ago, but gave up since there was a lot of resistance to handling the subtle complexities of these kind of problems. See discussion: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=449497#212 But anyway, I think to get anywhere you'll need to help get Debian policy 2.2.1 clarified for these kind of conditions. Then you'll be able to submit bugs with appropriate RC severity so they'll have to be handled. Best wishes, Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CANTw=MMFV5em3BZ8gX4LZabAL=grs2rswa3l0gdxj6-sjci...@mail.gmail.com
Re: leaks in our only-signed-software fortress
On Mon, 2012-02-20 at 09:56 +, Philipp Kern wrote: > Well, the rationale is documented in #333552 (which is linked to by the > changelog). I dropped some words on "rationale" to the aforementioned bug,... > AIUI it doesn't matter because it's just about randomizing > unused parts of the packet. Well as far as I can see it IS used per default by the packages... but as both maintainers asked me between the lines not to contribute anymore, I've just lost interest. The nagios/icinga/plugins packages are at least in some parts in a quite questionable state anyway. Happy wishes, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: leaks in our only-signed-software fortress
On 2012-02-20, Christoph Anton Mitterer wrote: > 2) Well I really feel bad now, having to point my finger at the Nagios > Maintainers as they really do a good job, but this just shocked me: > Bug #660585. > > Well as I describe in the bug, it is practically (at the moment) of no > relevance as SSL in Nagios' NRPE is broken. > The problem here is IMHO rather the attitude behind. Well, the rationale is documented in #333552 (which is linked to by the changelog). AIUI it doesn't matter because it's just about randomizing unused parts of the packet. Kind regards Philipp Kern -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/slrnjk466h.1o2.tr...@kelgar.0x539.de
Re: leaks in our only-signed-software fortress
On Sat, Feb 18, 2012 at 06:09:37AM +0200, Christoph Anton Mitterer wrote: > - packages that are just wrapper packages, download something from > somewhere without doing any > hashsum checks at all How many in main? -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120220094000.GC14643@debian
Re: leaks in our only-signed-software fortress
Hey. Just by now,... two additional cases of security problems crossed my mind. Though not related to our package signing, they originate to some extent in the same problem as everything that was discussed in this thread before: Insufficient "feeling" for security [by maintainers]. 1) Silent modification of defaults. IMHO, it's perfectly well for a maintainer, to modify the default _configuration_ of and even to "lower" security by that (to the later: if there is really good reason for it). The admin installing the package shall be expected to go through the configuration and understand it. If he doesn't it's his fault not the maintainers. But I've already seen several cases, where maintainers choose to directly modify (patch) the defaults in the program code. Sometimes this was for good reasons, sometimes I could not follow it (as in my mind, a simple change of the default config would have been enough). Nevertheless, in such a case it is utmost important that these modifications are documented. And not just in the quilt patch header. This should really go into several places: - any --help output of the program itself must be updated, too. - manpages/documentation should be added with information (in the respective sections) that Debian deviates from the upstream default value (and how). - There should be one central point, where all those modifications are _clearly_ (that is not hidden between words of text) documented (probably README.Debian). 2) Well I really feel bad now, having to point my finger at the Nagios Maintainers as they really do a good job, but this just shocked me: Bug #660585. Well as I describe in the bug, it is practically (at the moment) of no relevance as SSL in Nagios' NRPE is broken. The problem here is IMHO rather the attitude behind. Cheers, Chris. smime.p7s Description: S/MIME cryptographic signature
Re: leaks in our only-signed-software fortress
On Sat, Feb 18, 2012 at 04:42:38PM -0200, Henrique de Moraes Holschuh wrote: > > Against what? The source is only downloaded from upstream once per > > upstream release, what is there to check against? > > Upstream VCS, previous releases (when the diff is small enough), request > that upstream publish in an email message the sha1sum or sha256sum when they > announce a new release, etc. A good part of upstreams use git, let's educate them about signed tags. -- // If you believe in so-called "intellectual property", please immediately // cease using counterfeit alphabets. Instead, contact the nearest temple // of Amon, whose priests will provide you with scribal services for all // your writing needs, for Reasonable and Non-Discriminatory prices. signature.asc Description: Digital signature
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012, Philip Hands wrote: > On Sat, 18 Feb 2012 15:49:30 +, Neil Williams wrote: > > On Sat, 18 Feb 2012 16:25:20 +0200 > > Christoph Anton Mitterer wrote: > > > Am 18.02.2012 14:40, schrieb Neil Williams: > > > >> I think as a start it should be made a policy that any "wrapper" > > > >> package that > > > >> downloads code from the net must at least do a strong checksum check > > > >> on the > > > >> downloaded code. > > > > Not possible to enforce as a 'MUST' because, by definition, > > > > third-party > > > > websites will not provide checksums for every possible download > > > > mechanism. > > > > > > Well it's still possible then,... the maintainer can just calculate > > > sums on his own. > > > > Against what? The source is only downloaded from upstream once per > > upstream release, what is there to check against? > > He's talking about stuff like flash-nonfree (or whatever) where we ship > a wrapper that wgets the actual tarball for you from the distributor, > and checks the checksum of whatever it ends up with. > > The maintainer can grab a copy, checksum it (perhaps if paranoid do the > download from elsewhere on a different day, make sure the checksums > match), and then sign a package containing the checksum that he > generated to ensure that everyone that installs the package gets the > same tarball, or sees an error message. I believe there are "downloader" packages in Debian that do just that, and the practice isn't new. It is far better than nothing, as it effectively shortens the time window where a trojan can be inserted in the distribution point. That's not even close to 100% coverage, but it is nothing to sneeze at, either. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218184507.gi20...@khazad-dum.debian.net
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012, Neil Williams wrote: > On Sat, 18 Feb 2012 16:25:20 +0200 > Christoph Anton Mitterer wrote: > > Am 18.02.2012 14:40, schrieb Neil Williams: > > >> I think as a start it should be made a policy that any "wrapper" > > >> package that > > >> downloads code from the net must at least do a strong checksum check > > >> on the > > >> downloaded code. > > > Not possible to enforce as a 'MUST' because, by definition, > > > third-party > > > websites will not provide checksums for every possible download > > > mechanism. > > > > Well it's still possible then,... the maintainer can just calculate > > sums on his own. > > Against what? The source is only downloaded from upstream once per > upstream release, what is there to check against? Upstream VCS, previous releases (when the diff is small enough), request that upstream publish in an email message the sha1sum or sha256sum when they announce a new release, etc. How much it will protect Debian users, depends entirely where the trojan instertion point was. So far, the more common insertion points have NOT been upstream's development box, but rather the public distribution points and vcs trees. Heck, even for dead upstream you still get the stuff from the other distros to compare Debian's with, even if you did it only to check for interesting patches from other distros, it would still increase the chances of you noticing something is weird. And it is part of the job of a downstream maintainer to educate upstream when necessary, even if it takes a lot of diplomacy and a lot of effort. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218184237.gh20...@khazad-dum.debian.net
Re: leaks in our only-signed-software fortress
On 02/18/2012 09:30 PM, Josselin Mouette wrote: > Plugin integrity is > guaranteed by SSL, and extensions have been checked before being put on > the website. > I feel much much safer now that I know that my plugin downloads are protected by Diginotar ! :) Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f3febc7.2040...@debian.org
Re: leaks in our only-signed-software fortress
On 02/18/2012 08:40 PM, Neil Williams wrote: > On Sat, 18 Feb 2012 11:48:27 +0100 > Thomas Koch wrote: > > >> I think as a start it should be made a policy that any "wrapper" package >> that >> downloads code from the net must at least do a strong checksum check on the >> downloaded code. >> > Not possible to enforce as a 'MUST' because, by definition, third-party > websites will not provide checksums for every possible download > mechanism. > We're trying to mitigate risks of a man-in-the-middle attack here. Not to authenticate a content, which is the job of the maintainer. We want to check that the file is the same one as the one the maintainer downloaded. Which means that if there isn't a checksum on the third-party website, a maintainer can just run sha512sum and save the checksum in his download script (or next to it) by himself for later runtime check. So yes, a MUST can happen, IMO. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f3fea8b.50...@debian.org
Re: leaks in our only-signed-software fortress
On Sat, Feb 18, 2012 at 04:31:19PM +0100, Ansgar Burchardt wrote: > Jakub Wilk writes: > > * Ansgar Burchardt , 2012-02-18, 14:14: > >>>Could you point us to those which were ignored or denied? > >>At least pbuilder still disables secure APT by default, see #579028. > > > > The bug is closed. Am I missing something? > > pbuilder was changed to pass the --keyring argument to debootstrap by > default and there now is an option to enable secure apt, but it is still > disabled by default. This should be safe to enable now. We had problems enabling it in sbuild (in 2005) when tools first started supporting it for backward- compatibility reasons, but it's been the default for some time now (since July 2008) since everything we would want to build on supports it. Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linuxhttp://people.debian.org/~rleigh/ `. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools `-GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218174609.gi26...@codelibre.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 18:45, schrieb Philip Hands: He's talking about stuff like flash-nonfree (or whatever) where we ship a wrapper that wgets the actual tarball for you from the distributor, and checks the checksum of whatever it ends up with. Yes! (perhaps if paranoid do the download from elsewhere on a different day, make sure the checksums match), Actually things like this should be done, if nothing better (signatures + trust path) is available... of course it doesn't make things 100% sure, but even if it gets us just some 10% likeliness of noting an attack it's worth it (IMHO). Cheers, Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1f837ecb03fe0f56151a5e3c6369b...@scientia.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 19:03, schrieb brian m. carlson: On Sat, Feb 18, 2012 at 11:48:27AM +0100, Thomas Koch wrote: What about a debhelper script that receives an URL (or set of mirror URLs) and a SHA1 and does the download and check? Please use something stronger than SHA-1. SHA-1 has some weaknesses and something like SHA-256 or SHA-512 should be used in new applications. +1 SHA1 has some weaknesses but I guess it's not yet (!!!) something were we have to make big concerns in real world threads... but: I guess the main point with respect to things like these is the following: Maintainers (or people in general) SHOULD get a sense of security and the obvious question here is: Why use something weaker, when something better is broadly available and technically feasible (I mean e.g. sums on source code make no performance problem,... when someone does streaming of large data, there can be benefit from using something weaker but faster (e.g. MD5 or) in contrast to stronger but slower (SHA512). Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/b116548c250ddc4aba34ee83384c8...@scientia.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 16:18, schrieb Jakub Wilk:
The bug is closed. Am I missing something?
But anyway, this is saddening. Hundreds (? - wild guess) of
developers have been building their packages in insecure environment,
yet pbuilder maintainer and a member of Security Team believe that it
was a feature, not a bug. :|
And looking at my current sid pbuilderrc manpage I read at least:
APTGETOPT=('--force-yes')
Extra flags to give to apt-get. Default is --force-yes,
which
will skip key verification of packages to be installed.
Unset if
you want to enable key verification.
=> what does verification mean here?
and
PBUILDERSATISFYDEPENDSOPT=('--check-key')
Array of flags to give to pbuilder-satisfydepends.
Specifying
--check-key here will try to verify key signatures.
=> "try"? Doesn't sound trustworthy at least :(
Cheers,
Chris.
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/42cef275f2916e38bd6cb5c037dcc...@scientia.net
Re: leaks in our only-signed-software fortress
On Sat, Feb 18, 2012 at 11:48:27AM +0100, Thomas Koch wrote: > What about a debhelper script that receives an URL (or set of mirror > URLs) and a SHA1 and does the download and check? Please use something stronger than SHA-1. SHA-1 has some weaknesses and something like SHA-256 or SHA-512 should be used in new applications. -- brian m. carlson / brian with sandals: Houston, Texas, US +1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187 signature.asc Description: Digital signature
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012 15:49:30 +, Neil Williams wrote: > On Sat, 18 Feb 2012 16:25:20 +0200 > Christoph Anton Mitterer wrote: > > > Am 18.02.2012 14:40, schrieb Neil Williams: > > >> I think as a start it should be made a policy that any "wrapper" > > >> package that > > >> downloads code from the net must at least do a strong checksum check > > >> on the > > >> downloaded code. > > > Not possible to enforce as a 'MUST' because, by definition, > > > third-party > > > websites will not provide checksums for every possible download > > > mechanism. > > > > Well it's still possible then,... the maintainer can just calculate > > sums on his own. > > Against what? The source is only downloaded from upstream once per > upstream release, what is there to check against? He's talking about stuff like flash-nonfree (or whatever) where we ship a wrapper that wgets the actual tarball for you from the distributor, and checks the checksum of whatever it ends up with. The maintainer can grab a copy, checksum it (perhaps if paranoid do the download from elsewhere on a different day, make sure the checksums match), and then sign a package containing the checksum that he generated to ensure that everyone that installs the package gets the same tarball, or sees an error message. Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560]http://www.hands.com/ |-| HANDS.COM Ltd.http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND pgpfT82GLERlY.pgp Description: PGP signature
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012 15:59:38 +0200 Christoph Anton Mitterer wrote: > Am 18.02.2012 10:11, schrieb Teus Benschop: > > To put things in perspective, I just wonder how strong this > > 'fortress' > > really is, and whether this strength is only in our perception or > > whether it is real. Let me give just one example: A developer > > downloads > > a tarball from an upstream source, configures it, and does make > > install, > > yet has not even once checked whether this tarball is secure or is > > not a > > root kit. Detecting a root kit in downloaded source isn't trivial. If the package has completely changed and has been *replaced* with a toolkit, then any maintainer will notice that it doesn't build the files expected. A determined attack which tries to embed a root kit within an existing package/binary would be much more difficult for anyone to pick up, potentially even upstream. If a binary file built for the previous version of the upstream is a few Kb bigger in the next release, no maintainer is going to see that as anything more than entirely expected of a new upstream release. > This is true but... > a) this would be a general attack against all people, which are usually > a tiny bit harder to do, then the local sysadmin just hacking > colleagues.. So what is the point of the entire thread? Downloading upstream sources is a general thing for everyone using that package but it is done by one person (usually) and (usually) without any upstream checksums. It's a complete non-issue because there is 0% chance of getting every conceivable upstream team to implement anything like this. There may be other clues (file naming conventions, layout styles, weird compile messages appearing etc.) but none of those are reliable. > b) as everyone is affected then (all users of the package),... there is > a greater chance of notifying it But it is still down to chance. Overall, it's not something that I'm going to lose any sleep over. There's no way that I'll be asking my upstream teams to even consider it, nor would I implement it for my own upstream projects. I happen to use upstream sites which provide something like this but their security models aren't perfect. I used to provide detached GnuPG signatures alongside my uploaded source tarballs but nobody cared or even noticed if I inadvertently broke the signature. (This is for packages which regularly got downloaded for inclusion into Fedora, ArchLinux, Gentoo and numerous other distros other than Debian/Debian-based ones which get the source directly from me.) Honestly, nobody cares. > most important... > c) the ideal situation would of course be, that the maintainer has a > good relationship to upstream, perhaps even met them in person, > exchanged OpenPGP keys with them and uses those (or weaker means[0]) to > verify every single download. Completely unrealistic for most maintainers. Now, I could be really smug here and say that for some of my packages, I have absolute and airtight security between maintainer and upstream but that's only because I am sole maintainer and sole upstream.. (if you want better security than that, keep dreaming) .. but I also have other packages where upstream provide no checksums whatsoever. I've been lucky enough to meet one member of one of those upstream teams once (at a conference ~3 yrs ago which I haven't been able to attend since) but we had much more important stuff to discuss than signing / checksumming the download directory. Besides, meeting someone once with little chance of ever meeting them again is not a particularly strong indication that I should sign their GnuPG key (which is why I don't attend mass keysignings any longer). -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpSoIf9ctVJ1.pgp Description: PGP signature
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012 16:25:20 +0200 Christoph Anton Mitterer wrote: > Am 18.02.2012 14:40, schrieb Neil Williams: > >> I think as a start it should be made a policy that any "wrapper" > >> package that > >> downloads code from the net must at least do a strong checksum check > >> on the > >> downloaded code. > > Not possible to enforce as a 'MUST' because, by definition, > > third-party > > websites will not provide checksums for every possible download > > mechanism. > > Well it's still possible then,... the maintainer can just calculate > sums on his own. Against what? The source is only downloaded from upstream once per upstream release, what is there to check against? -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpV0KWdtHDkd.pgp Description: PGP signature
Re: leaks in our only-signed-software fortress
Jakub Wilk writes: > * Ansgar Burchardt , 2012-02-18, 14:14: >>>Could you point us to those which were ignored or denied? >>At least pbuilder still disables secure APT by default, see #579028. > > The bug is closed. Am I missing something? pbuilder was changed to pass the --keyring argument to debootstrap by default and there now is an option to enable secure apt, but it is still disabled by default. Regards, Ansgar -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zkcgb1ko@deep-thought.43-1.org
Re: leaks in our only-signed-software fortress
* Christoph Anton Mitterer , 2012-02-18, 16:19: Take the non-free flash as example... (yeah I know it's non-free and not officially sec-supported).. Even if it would use some SHA512 sums (hardcoded into the package) to verify the download (I don't know whether it does),.. the update mechanism is still outsite of the package management system (on has to call update-flash or something like that)... so you bypass the whole central point of update management. Completely agreed! We should remove flashplugin-nonfree from the archive. Or wait, even simpler, you could just not install it. FWIW, the Contents files _are_ signed, but AFAICS apt-file doesn't verify the signature. See #515942. You can easily check yourself that Contents-* checksums are mentioned in the Release files, even for lenny. (Though indeed they weren't in Feb 2009.) Feel free to unarchive and reopen the bug. -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218144754.ga6...@jwilk.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 15:30, schrieb Josselin Mouette: Personally I decided to use GNOME-fallback, but via the meta-packages I still got the GNOME shell... today I've noticed that it silently installs an extension, which (I can only assume this by the little description) does some software installation/enabling for GNOME shell from extensions.gnome.org. To me this sounds more like a root-kit than a feature. No GNOME shell extension is ever downloaded without your consent. The browser plugin is only here to make this possible. Plugin integrity is guaranteed by SSL, and extensions have been checked before being put on the website. Well I guess the problem here are three things: - Communication You say now, that GNOME checks all what they put up there, and nothing is every installed automatically. This makes things a bit better,... but it's not really obviously documented. At least not for a just-a-user like me. Of course one can always say read the code + go into the developer docs,... but if I have to do this for everything, than I'm just screwed. - Trust I really do not trust GNOME/Mozilla etc. here do do all this in a secure and right way. At least for Mozilla there are hundredths of extensions, they surely can't check them all. - Bypassing the package management system IMHO, software in Debian should ONLY be installed by the package management system with one exception: When the user really downloads/(optionally compiles)/installs himself. Especially software should not bring its own package management system in form of app-store-like thingies. Of course I know it's difficult to prevent this. Upstreams just do it... and ways around it (e.g. our Mozilla Extension Packages) are a big effort for us. Nevertheless, solve this via packages, would be the right way (IMHO). Anyway this doesn’t work very well so we’d be better with just putting those extensions in another Debian package, but I see this more as a functional problem than a security one. Great :) Cheers, Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/74ac1eea31fb134b60c8ef405d676...@scientia.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 14:40, schrieb Neil Williams: I think as a start it should be made a policy that any "wrapper" package that downloads code from the net must at least do a strong checksum check on the downloaded code. Not possible to enforce as a 'MUST' because, by definition, third-party websites will not provide checksums for every possible download mechanism. Well it's still possible then,... the maintainer can just calculate sums on his own. Of course this does not mean things are secure (the maintainer could already use a forged version)... but at least it helps again single MITM attacks. Cheers, Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/0a9d69dc96c647151114bca2d8ebb...@scientia.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 14:34, schrieb Neil Williams: >- packages that eventually run some code which was downloaded >unsecured. >debootstrap used to be like that, pbuilder, and some others Only a bug if this happens by default. It is perfectly acceptable to support an option to disable SecureApt - just as long as this is not the default. Tools in Debian need to work with systems outside Debian and those do not necessarily *need* SecureApt because the entire loop is internal or even local to the one machine. Agreed, but it WAS the default till recently,.. e.g. in debootstrap till 1.0.30, when my bug #560038 was fixed (thanks Joey :) ). And of course anything that used debootstrap (e.g. pbuilder, piuparts do so) was automatically insecure, too. (till then) Cheers, Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/121005584832f4d35086604441f21...@scientia.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 13:32, schrieb Jakub Wilk: I'll add to the list: - Packages that download and run untrusted code at build time. May I add a similar case... Take the non-free flash as example... (yeah I know it's non-free and not officially sec-supported).. Even if it would use some SHA512 sums (hardcoded into the package) to verify the download (I don't know whether it does),.. the update mechanism is still outsite of the package management system (on has to call update-flash or something like that)... so you bypass the whole central point of update management. FWIW, the Contents files _are_ signed, but AFAICS apt-file doesn't verify the signature. See #515942. But why is that a big deal? What do you mean? Of not verifying it? Well as always someone can attack you if you somehow (for whatever reason) rely on the information being correct. Moreover, if there is some automatic parsing of those files, you can also easily think of attack vectors by manipulating files,.. Could you point us to those which were ignored or denied? Phew... would have to do a lot of digging in my mails and bug reports to find them out again. Cheers, Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1ef6a4f196a3cdd6cab0b5940cbf4...@scientia.net
Re: leaks in our only-signed-software fortress
* Ansgar Burchardt , 2012-02-18, 14:14: Could you point us to those which were ignored or denied? At least pbuilder still disables secure APT by default, see #579028. The bug is closed. Am I missing something? But anyway, this is saddening. Hundreds (? - wild guess) of developers have been building their packages in insecure environment, yet pbuilder maintainer and a member of Security Team believe that it was a feature, not a bug. :| -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218141858.ga5...@jwilk.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 13:14, schrieb Benjamin Drung: This is no problem for us, because the malware was distributed on some untrustworthy websites. We, as packagers, get the code directly from the Videolan project. So you meet them once in person and exchanged some kind of PKI/shared secret etc? That's great then of course and the ideal case of securely getting the sources as a maintainer :-) But I guess only a small fraction of our packages have such a secure trust path to their upstream. Chris. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/c6728367706314e815f09031fcedd...@scientia.net
Re: leaks in our only-signed-software fortress
Am 18.02.2012 10:11, schrieb Teus Benschop: To put things in perspective, I just wonder how strong this 'fortress' really is, and whether this strength is only in our perception or whether it is real. Let me give just one example: A developer downloads a tarball from an upstream source, configures it, and does make install, yet has not even once checked whether this tarball is secure or is not a root kit. This is true but... a) this would be a general attack against all people, which are usually a tiny bit harder to do, then the local sysadmin just hacking colleagues.. b) as everyone is affected then (all users of the package),... there is a greater chance of notifying it most important... c) the ideal situation would of course be, that the maintainer has a good relationship to upstream, perhaps even met them in person, exchanged OpenPGP keys with them and uses those (or weaker means[0]) to verify every single download. Cheers, Chris. [0] Some projects secure their sites e.g. with X.509 certs by one of the commercial CAs I guess this is better than nothing, but many recent cases have shown us that the whole strict hierarchical trust model by X.509 is basically for trash. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/29f4cc7dc550446fc47e078c3c727...@scientia.net
Re: leaks in our only-signed-software fortress
Jakub Wilk writes: > Could you point us to those which were ignored or denied? At least pbuilder still disables secure APT by default, see #579028. Regards Ansgar -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/8762f4cmh1@deep-thought.43-1.org
Re: leaks in our only-signed-software fortress
Le samedi 18 février 2012 à 06:09 +0200, Christoph Anton Mitterer a écrit : > Personally I decided to use GNOME-fallback, but via the meta-packages I > still got the GNOME shell... today > I've noticed that it silently installs an extension, which (I can only > assume this by the little > description) does some software installation/enabling for GNOME shell > from extensions.gnome.org. > To me this sounds more like a root-kit than a feature. No GNOME shell extension is ever downloaded without your consent. The browser plugin is only here to make this possible. Plugin integrity is guaranteed by SSL, and extensions have been checked before being put on the website. Anyway this doesn’t work very well so we’d be better with just putting those extensions in another Debian package, but I see this more as a functional problem than a security one. -- .''`. Josselin Mouette : :' : `. `' `- signature.asc Description: This is a digitally signed message part
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012, Teus Benschop wrote: > To put things in perspective, I just wonder how strong this 'fortress' > really is, and whether this strength is only in our perception or > whether it is real. Let me give just one example: A developer downloads > a tarball from an upstream source, configures it, and does make install, > yet has not even once checked whether this tarball is secure or is not a > root kit. Teus. Good packaging developers go to great lengths to be sure they are not going to distribute anything trojaned. This takes a lot of work, and often requires very goot working relationship with upstream to the point of getting upstream to change his processes. This does include tracking deviations from VCS to upstream releases, going over upstream changes when possible, and using crypto properly to verify authenticity of upstream commits and tarballs (when available. When it is not available, educating upstream about it is required). Obviously, sometimes due diligence is not done (some people are quite lazy), and sometimes it is just plain impossible to do. And sometimes the malicious change was done in such way that only a careful audit would find it. So, yes, you really risk it happening. If you want to minimize that chance, Debian stable is your friend as the window of opportunity to discover trojaned sources is much larger in stable than it is in testing and unstable. I'm not sure what the Debian project could do to make sure we're at least doing everything that is humanly possible, *on every package* (we already do it on many packages) to allow for early detection of trojaned upstream releases or trojaned upstream VCS. -- "One disk to rule them all, One disk to find them. One disk to bring them all and in the darkness grind them. In the Land of Redmond where the shadows lie." -- The Silicon Valley Tarot Henrique Holschuh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218124650.gc20...@khazad-dum.debian.net
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012 11:48:27 +0100 Thomas Koch wrote: > I think as a start it should be made a policy that any "wrapper" package that > downloads code from the net must at least do a strong checksum check on the > downloaded code. Not possible to enforce as a 'MUST' because, by definition, third-party websites will not provide checksums for every possible download mechanism. Only a should is possible here - wherever a checksum can be verified independently, but even then, an unreliable upstream checksum method is better ignored than supported. > What about a debhelper script that receives an URL (or set of mirror URLs) > and > a SHA1 and does the download and check? Do you mean dget? If it's mirror URL's, most of the time you can use apt. If it's mirrors, fine. Anything downloaded from a site which is not part of *.debian.org or a mirror of *.debian.org cannot be expected to provide useful checksums. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpHgJMd2RLfl.pgp Description: PGP signature
Re: leaks in our only-signed-software fortress
On Sat, 18 Feb 2012 12:32:14 +0100 Jakub Wilk wrote: > * Christoph Anton Mitterer , 2012-02-18, 06:09: > >I've decided that I think it's important to CC this d-d: > >Debian has a good system of securing packages and making sure that only > >signed stuff comes to the user. > >Over time I've seen many holes in this: > >- packages that are just wrapper packages, download something from > >somewhere without doing any hashsum checks at all > >Some firmware packages, some font packages, documentation etc. is/was > >like that. > >- packages that eventually run some code which was downloaded > >unsecured. > >debootstrap used to be like that, pbuilder, and some others Only a bug if this happens by default. It is perfectly acceptable to support an option to disable SecureApt - just as long as this is not the default. Tools in Debian need to work with systems outside Debian and those do not necessarily *need* SecureApt because the entire loop is internal or even local to the one machine. > All(/most?) of those would be RC bugs. > I'll add to the list: > - Packages that download and run untrusted code at build time. ...if on Debian buildds or by default. Private buildd's, by a selectable option - not a bug. -- Neil Williams = http://www.linux.codehelp.co.uk/ pgpS3ax3E5EVf.pgp Description: PGP signature
Re: leaks in our only-signed-software fortress
* Christoph Anton Mitterer , 2012-02-18, 06:09: I've decided that I think it's important to CC this d-d: Debian has a good system of securing packages and making sure that only signed stuff comes to the user. Over time I've seen many holes in this: - packages that are just wrapper packages, download something from somewhere without doing any hashsum checks at all Some firmware packages, some font packages, documentation etc. is/was like that. - packages that eventually run some code which was downloaded unsecured. debootstrap used to be like that, pbuilder, and some others All(/most?) of those would be RC bugs. I'll add to the list: - Packages that download and run untrusted code at build time. - Some packages load and process content that could be secured but (is/was) not. IIRC the Contents Files are not signed and therefore e.g. apt-file cannot secure this. FWIW, the Contents files _are_ signed, but AFAICS apt-file doesn't verify the signature. But why is that a big deal? Of those who actually DID checks, there were several that used weak checks (even though there was no need to),... e.g. things like MD5 checks instead of something "better". For many of those I've reported bugs (and I'm sure I didn't found a lot of them, and I'm further sure that new cases were introduced). Some where closed, some where just ignored or denied. Could you point us to those which were ignored or denied? -- Jakub Wilk -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120218113214.ga2...@jwilk.net
Re: leaks in our only-signed-software fortress
Am Samstag, den 18.02.2012, 11:48 +0100 schrieb Thomas Koch: > July 2011 VLC suffers from Companies spreading Malware bundled with VLC This is no problem for us, because the malware was distributed on some untrustworthy websites. We, as packagers, get the code directly from the Videolan project. -- Benjamin Drung Debian & Ubuntu Developer signature.asc Description: This is a digitally signed message part
Re: leaks in our only-signed-software fortress
Christoph Anton Mitterer: > Hey. > > I've decided that I think it's important to CC this d-d: > Debian has a good system of securing packages and making sure that only > signed stuff comes to the user. > Over time I've seen many holes in this: > - packages that are just wrapper packages, download something from > somewhere without doing any >hashsum checks at all I'm as concerned as you are and collected some examples over time that might give a perspective: http://www.koch.ro/blog/index.php?/archives/153-On-distributing-binaries.html September 2009 Apache.org got hacked - twice in eight months. December 2010 The proftpd Source Code contains a backdoor January 2011 Sourceforge, one of the biggest distributor of free software, got hacked. June 2011 The Wordpress Plugins AddThis, WPtouch and W3 Total Cache contain backdoors July 2011 The vsftpd server download was replaced with a hacked version July 2011 VLC suffers from Companies spreading Malware bundled with VLC August 2011 kernel.org got hacked September 2011 MySQL.com hacked to server malware November 2011 Takedown of the largest botnet ever. DNS resolving of the bots was compromised. December 2011 Does download.com enrich their downloads with malware? February 2012 unnoticed for 3 months, the Horde project served compromised downloads I think as a start it should be made a policy that any "wrapper" package that downloads code from the net must at least do a strong checksum check on the downloaded code. What about a debhelper script that receives an URL (or set of mirror URLs) and a SHA1 and does the download and check? Regards, Thomas Koch, http://www.koch.ro signature.asc Description: This is a digitally signed message part.
Re: leaks in our only-signed-software fortress
To put things in perspective, I just wonder how strong this 'fortress' really is, and whether this strength is only in our perception or whether it is real. Let me give just one example: A developer downloads a tarball from an upstream source, configures it, and does make install, yet has not even once checked whether this tarball is secure or is not a root kit. Teus. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f3f5d25.3000...@gmail.com
leaks in our only-signed-software fortress
Hey.
I've decided that I think it's important to CC this d-d:
Debian has a good system of securing packages and making sure that only
signed stuff comes to the user.
Over time I've seen many holes in this:
- packages that are just wrapper packages, download something from
somewhere without doing any
hashsum checks at all
Some firmware packages, some font packages, documentation etc. is/was
like that.
- packages that eventually run some code which was downloaded
unsecured.
debootstrap used to be like that, pbuilder, and some others
- Some packages load and process content that could be secured but
(is/was) not.
IIRC the Contents Files are not signed and therefore e.g. apt-file
cannot secure this.
Of those who actually DID checks, there were several that used weak
checks (even though there was no
need to),... e.g. things like MD5 checks instead of something "better".
For many of those I've reported bugs (and I'm sure I didn't found a lot
of them, and I'm further sure
that new cases were introduced).
Some where closed, some where just ignored or denied.
Recently with the Web2.0 and AppStore/Marked/etc. hype, things got even
worse.
I've you wanna be cool, you cannot just distribute software that people
can get and install regularly.
You need some AppStore, where no user has any controll on
sources/security/etc. often you even cannot
control when updates happen.
Mainly via the browsers (Mozilla, Chromium) this shit has found it's
way into Debian.
Software installation bypasses the Debian archives but comes directly
from any internet source
and get's installed locally in the user's homedir.
For Firefox we have fortunately a good team, which does real packagin
of the extensions and the plugins.
And Mike and the FF maintainers do a good job in trying to integrate
Mozilla's extension crap into the
Debian way.
Nevertheless there are still holes from time to time,.. e.g. that FF
tried to update extensions installed
from a Debian package.
Now the GNOME guys (talking about upstream) seem to be the new kids at
the sandbox and when the've decided
to assimilate the world with GNOME shell they also needed kind of an
app store, I guess.
See my bug #660311.
Personally I decided to use GNOME-fallback, but via the meta-packages I
still got the GNOME shell... today
I've noticed that it silently installs an extension, which (I can only
assume this by the little
description) does some software installation/enabling for GNOME shell
from extensions.gnome.org.
To me this sounds more like a root-kit than a feature.
This rant is not (!!) about blaming our GNOME maintainers, who really
do some good job, ... I just hope to start
some discussion about how Debian should deal with such hype
developments ("Apps") which may be nice
for users, but not for security.
And also about the other mentioned "holes" in our beloved fortress that
allows only signed code
to get onto the system (unless of course you install something manually
:P)
I mean there would be many places in Debian, where security could be
improved... webservers shouldn't need a
fancy default-works-out-of-the-box config which displays some Hello
World pages... and actually, IMHO installing
a daemon should not mean that it's automatically enabled (speaking of
init scripts)... the config is likely
not yet finished/secured.
Well I doubt that things will change there... but we really should take
care on whom we allow to provide our
users with "external" software. Especially when this happens easily
without the control (or much interaction)
of the users and/or admin.
Cheers,
Chris.
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/0763775752c72b696283491568e04...@scientia.net
