Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
Karsten M. Self wrote: > > It had to be re-installed. You probably know that since you've read > > the announcement we were able to send out before the machine was taken > > down for reinstallation. > > That announcement wasn't delivered for all users until _after_ murphy > was resurrected. I myself got the debian-security-announce message > mailed Nov 21 on 25 Nov 2003 15:16:56 -0800. That's true since murphy was powered down for a re-install in the middle of its delivery. The (same) mail on debian-announce should have been delivered by that time. Regards, Joey -- Have you ever noticed that "General Public Licence" contains the word "Pub"?
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
Karsten M. Self wrote: > > It had to be re-installed. You probably know that since you've read > > the announcement we were able to send out before the machine was taken > > down for reinstallation. > > That announcement wasn't delivered for all users until _after_ murphy > was resurrected. I myself got the debian-security-announce message > mailed Nov 21 on 25 Nov 2003 15:16:56 -0800. That's true since murphy was powered down for a re-install in the middle of its delivery. The (same) mail on debian-announce should have been delivered by that time. Regards, Joey -- Have you ever noticed that "General Public Licence" contains the word "Pub"? -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
On Fri, Nov 28, 2003 at 04:14:19AM -0800, Karsten M. Self wrote: > I'll disagree with Martin's comment that the server compromise didn't > constitute a security issue despite the lack of an archive compromise. > For someone well versed in Debian procedures, it might have been > plausible that the archives themselves weren't compromised. For a > typical user, I don't think this was the case. For the typical user's > management or clients, it's very likely _not_ the case, and a timely > positive statement of status would be very, very helpful. > > Security affecting Debian servers _potentially_ affects Debian packages. > As it was, I cleared my locale package cache and stopped updates on > hearing about the compromise. It wasn't for another few hours that I > was aware that the archive was reportedly _not_ compromised. > > In the absense of any information, the security status of Debian project > packages in the event of a known or rumored server compromise is at best > unknown. It wasn't clear to me that the packages that I had downloaded were safe, and it even wasn't clear after reading that the archives were safe. I suggest some phrase like "packages in the debian archive" or just "debian packages." The reason is that "archive" usually means something covering (ancient) history. I initially thought it referred to the mailing list archives. If I'd thought harder, I might have thought it referred to past debian packages (which I think are provided via snapshot.debian.org?? I've never used them). Perhaps I should have known better, but since the confusion seems pretty easy, and pretty easy to fix, I suggest fixing it if we should ever have such an unfortunate incident again. Thanks to all those who worked so hard to detect, and then correct, this problem. Ross Boylan
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
On Fri, Nov 28, 2003 at 04:14:19AM -0800, Karsten M. Self wrote: > I'll disagree with Martin's comment that the server compromise didn't > constitute a security issue despite the lack of an archive compromise. > For someone well versed in Debian procedures, it might have been > plausible that the archives themselves weren't compromised. For a > typical user, I don't think this was the case. For the typical user's > management or clients, it's very likely _not_ the case, and a timely > positive statement of status would be very, very helpful. > > Security affecting Debian servers _potentially_ affects Debian packages. > As it was, I cleared my locale package cache and stopped updates on > hearing about the compromise. It wasn't for another few hours that I > was aware that the archive was reportedly _not_ compromised. > > In the absense of any information, the security status of Debian project > packages in the event of a known or rumored server compromise is at best > unknown. It wasn't clear to me that the packages that I had downloaded were safe, and it even wasn't clear after reading that the archives were safe. I suggest some phrase like "packages in the debian archive" or just "debian packages." The reason is that "archive" usually means something covering (ancient) history. I initially thought it referred to the mailing list archives. If I'd thought harder, I might have thought it referred to past debian packages (which I think are provided via snapshot.debian.org?? I've never used them). Perhaps I should have known better, but since the confusion seems pretty easy, and pretty easy to fix, I suggest fixing it if we should ever have such an unfortunate incident again. Thanks to all those who worked so hard to detect, and then correct, this problem. Ross Boylan -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
On Fri, Nov 28, 2003 at 01:52:14PM +0100, Kjetil Kjernsmo wrote: > I learnt on /. that it had been a password compromise, so that meant, it > was in the generic class of problems. We're always vulnerable towards > that. But, we're all likely to be vulnerable to the local exploit used > to gain root. Besides, it was /. :-) >From the report I just read, sniffed password compromise to get in... but an as yet unknown privilege escalation from user to root once on board. -- -- Dale Amon [EMAIL PROTECTED]+44-7802-188325 International linux systems consultancy Hardware & software system design, security and networking, systems programming and Admin "Have Laptop, Will Travel" --
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
On Fri, Nov 28, 2003 at 01:52:14PM +0100, Kjetil Kjernsmo wrote: > I learnt on /. that it had been a password compromise, so that meant, it > was in the generic class of problems. We're always vulnerable towards > that. But, we're all likely to be vulnerable to the local exploit used > to gain root. Besides, it was /. :-) >From the report I just read, sniffed password compromise to get in... but an as yet unknown privilege escalation from user to root once on board. -- -- Dale Amon [EMAIL PROTECTED]+44-7802-188325 International linux systems consultancy Hardware & software system design, security and networking, systems programming and Admin "Have Laptop, Will Travel" -- -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
On Friday 28 November 2003 13:14, Karsten M. Self wrote: >That announcement wasn't delivered for all users until _after_ murphy >was resurrected. I myself got the debian-security-announce message >mailed Nov 21 on 25 Nov 2003 15:16:56 -0800. Hm, I got that late too, but the (unsigned) announcement got to debian-announce before the takedown. > First I want to say that the Debian project, in extremely adverse > circumnstances, comported itself well, disseminated information, if > not fully effectively, well beyond its nominal capacity with both web > and email services offline. Disclosures were timely, informative, > and helpful, while restraining themselves to established facts and > working within constraints of an as yet ongoing investigation. Very > few organizations can claim as much. Not only this, but it appears > at this point that the crown jewels -- the Debian archives and > mirrored distribution points themselves -- were _not_ compromised. > Commendable. Absolutely! > I'll disagree with Martin's comment that the server compromise didn't > constitute a security issue despite the lack of an archive > compromise. > Security affecting Debian servers _potentially_ affects Debian > packages. Yes, and I think the point needs emphasis that even if the archives are not compromised, what has happened to the Debian servers is very relevant to the security of all Debian users. My first thought when heared about the compromise was "ouch, that probably means, I'm vulnerable too". I considered for a moment to take my main server offline. The problem is of course that we all run the much of the same software that is on the Debian machines. Unless there are something generic that is a known problem (such as a sniffed password), or something that is special to one of the servers (e.g. BTS), the attacker might be able to use the attack he used on the Debian servers on pretty much _any_ Debian box. That's really scary. I learnt on /. that it had been a password compromise, so that meant, it was in the generic class of problems. We're always vulnerable towards that. But, we're all likely to be vulnerable to the local exploit used to gain root. Besides, it was /. :-) For these reasons, I think it is fair to say that any compromise on the Debian servers is very relevant to the security of all users. And that was the information I was missing earlier, to what extent I would myself be vulnerable. Also, I'm not a regular IRC user, so it didn't occur to me at the time that it was an alternative for gathering information. Besides, how is it with signatures on IRC? Best, Kjetil -- Kjetil Kjernsmo Astrophysicist/IT Consultant/Skeptic/Ski-orienteer/Orienteer/Mountaineer [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Homepage: http://www.kjetil.kjernsmo.net/OpenPGP KeyID: 6A6A0BBC
Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
on Wed, Nov 26, 2003 at 09:30:05AM +0100, Martin Schulze ([EMAIL PROTECTED]) wrote: > Dan Jacobson wrote: > > To us debian users, the most notable thing during this break in or > > whatever episode, is how the communication structures crumbled. > > It had to be re-installed. You probably know that since you've read > the announcement we were able to send out before the machine was taken > down for reinstallation. That announcement wasn't delivered for all users until _after_ murphy was resurrected. I myself got the debian-security-announce message mailed Nov 21 on 25 Nov 2003 15:16:56 -0800. > > debian-announce had one message on the 21st, five days ago, saying for > > more information, see www.debian.org. > > You'll find the same information linked on the front-page. Since the > web infrastructure was affected as well, but you already knew that > since it was mentioned in the announcement, it was not that easy > updating the web server. However, after a day we finally managed to > do that. > > > Nothing special there, so I checked http://www.debian.org/security/, > > same problem. > > As you know http://www.debian.org/security/ if for security > announcements regarding the packages Debian distributes. It has > nothing to do with the security on the Debian machines. Hence, it's > the wrong place. First I want to say that the Debian project, in extremely adverse circumnstances, comported itself well, disseminated information, if not fully effectively, well beyond its nominal capacity with both web and email services offline. Disclosures were timely, informative, and helpful, while restraining themselves to established facts and working within constraints of an as yet ongoing investigation. Very few organizations can claim as much. Not only this, but it appears at this point that the crown jewels -- the Debian archives and mirrored distribution points themselves -- were _not_ compromised. Commendable. Some bits could be improved, which is what I'm focusing on below. I'll disagree with Martin's comment that the server compromise didn't constitute a security issue despite the lack of an archive compromise. For someone well versed in Debian procedures, it might have been plausible that the archives themselves weren't compromised. For a typical user, I don't think this was the case. For the typical user's management or clients, it's very likely _not_ the case, and a timely positive statement of status would be very, very helpful. Security affecting Debian servers _potentially_ affects Debian packages. As it was, I cleared my locale package cache and stopped updates on hearing about the compromise. It wasn't for another few hours that I was aware that the archive was reportedly _not_ compromised. In the absense of any information, the security status of Debian project packages in the event of a known or rumored server compromise is at best unknown. Communications in an emergency sitation is paramount, and a number of people clearly _didn't_ get informed through back channels. I myself was _on_ IRC as word started leaking out, and still wasn't fully certain of what was going on or what to trust. Wichert's website (which I only learned was his the 27th!) was very helpful, as was the coverage provided by Slashdot and elsewhere. Discussion this with Manoj on IRC, my suggestion as summarized by him is that Debian should have an emergency response plan, part of which is a communications policy in the event a similar future compromise or systems failure. Specifically: - Triggering events. There are thresholds below which notifications needn't be triggered, and above which they very much should. Suggested: any event significantly affecting perceptions of security of the Debian archives or servers. Any outage of mail, web, or archive services anticipated to last beyond . E.g.: 6-12 hours, across core servers (but not mirrors). Any core server root compromise. *Not* single-package issues. Nuclear war or asteroid strike: you're on your own. - Where to provide information. Personal websites and news channels served well, but an advance statement of "here's where you should turn in the event of an emergency" would be useful. - What information to provide. Specifically, - the known (or unknown) status of archive or package compromise. - diagnostic checks; and/or - cleanup procedures. Wichert's pages on this would be a good template. By "known (or unkown)", I mean: if the archives are reasonably known to be safe, or are known to be compromised, this is communicated. If an assessment cannot be made with confidence, _that_ fact should be stated, e.g.: "the current security of the archives is unknown". By diagnostics and cleanup: pointers to tools or documentation explaining how to assess and/or secure a system. Wipe and rebuild if necessary. Again, wiggy.net
Re: Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
On Friday 28 November 2003 13:14, Karsten M. Self wrote: >That announcement wasn't delivered for all users until _after_ murphy >was resurrected. I myself got the debian-security-announce message >mailed Nov 21 on 25 Nov 2003 15:16:56 -0800. Hm, I got that late too, but the (unsigned) announcement got to debian-announce before the takedown. > First I want to say that the Debian project, in extremely adverse > circumnstances, comported itself well, disseminated information, if > not fully effectively, well beyond its nominal capacity with both web > and email services offline. Disclosures were timely, informative, > and helpful, while restraining themselves to established facts and > working within constraints of an as yet ongoing investigation. Very > few organizations can claim as much. Not only this, but it appears > at this point that the crown jewels -- the Debian archives and > mirrored distribution points themselves -- were _not_ compromised. > Commendable. Absolutely! > I'll disagree with Martin's comment that the server compromise didn't > constitute a security issue despite the lack of an archive > compromise. > Security affecting Debian servers _potentially_ affects Debian > packages. Yes, and I think the point needs emphasis that even if the archives are not compromised, what has happened to the Debian servers is very relevant to the security of all Debian users. My first thought when heared about the compromise was "ouch, that probably means, I'm vulnerable too". I considered for a moment to take my main server offline. The problem is of course that we all run the much of the same software that is on the Debian machines. Unless there are something generic that is a known problem (such as a sniffed password), or something that is special to one of the servers (e.g. BTS), the attacker might be able to use the attack he used on the Debian servers on pretty much _any_ Debian box. That's really scary. I learnt on /. that it had been a password compromise, so that meant, it was in the generic class of problems. We're always vulnerable towards that. But, we're all likely to be vulnerable to the local exploit used to gain root. Besides, it was /. :-) For these reasons, I think it is fair to say that any compromise on the Debian servers is very relevant to the security of all users. And that was the information I was missing earlier, to what extent I would myself be vulnerable. Also, I'm not a regular IRC user, so it didn't occur to me at the time that it was an alternative for gathering information. Besides, how is it with signatures on IRC? Best, Kjetil -- Kjetil Kjernsmo Astrophysicist/IT Consultant/Skeptic/Ski-orienteer/Orienteer/Mountaineer [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Homepage: http://www.kjetil.kjernsmo.net/OpenPGP KeyID: 6A6A0BBC -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Improved Debian Project Emergency Communications (was Re: communication structures crumbled)
on Wed, Nov 26, 2003 at 09:30:05AM +0100, Martin Schulze ([EMAIL PROTECTED]) wrote: > Dan Jacobson wrote: > > To us debian users, the most notable thing during this break in or > > whatever episode, is how the communication structures crumbled. > > It had to be re-installed. You probably know that since you've read > the announcement we were able to send out before the machine was taken > down for reinstallation. That announcement wasn't delivered for all users until _after_ murphy was resurrected. I myself got the debian-security-announce message mailed Nov 21 on 25 Nov 2003 15:16:56 -0800. > > debian-announce had one message on the 21st, five days ago, saying for > > more information, see www.debian.org. > > You'll find the same information linked on the front-page. Since the > web infrastructure was affected as well, but you already knew that > since it was mentioned in the announcement, it was not that easy > updating the web server. However, after a day we finally managed to > do that. > > > Nothing special there, so I checked http://www.debian.org/security/, > > same problem. > > As you know http://www.debian.org/security/ if for security > announcements regarding the packages Debian distributes. It has > nothing to do with the security on the Debian machines. Hence, it's > the wrong place. First I want to say that the Debian project, in extremely adverse circumnstances, comported itself well, disseminated information, if not fully effectively, well beyond its nominal capacity with both web and email services offline. Disclosures were timely, informative, and helpful, while restraining themselves to established facts and working within constraints of an as yet ongoing investigation. Very few organizations can claim as much. Not only this, but it appears at this point that the crown jewels -- the Debian archives and mirrored distribution points themselves -- were _not_ compromised. Commendable. Some bits could be improved, which is what I'm focusing on below. I'll disagree with Martin's comment that the server compromise didn't constitute a security issue despite the lack of an archive compromise. For someone well versed in Debian procedures, it might have been plausible that the archives themselves weren't compromised. For a typical user, I don't think this was the case. For the typical user's management or clients, it's very likely _not_ the case, and a timely positive statement of status would be very, very helpful. Security affecting Debian servers _potentially_ affects Debian packages. As it was, I cleared my locale package cache and stopped updates on hearing about the compromise. It wasn't for another few hours that I was aware that the archive was reportedly _not_ compromised. In the absense of any information, the security status of Debian project packages in the event of a known or rumored server compromise is at best unknown. Communications in an emergency sitation is paramount, and a number of people clearly _didn't_ get informed through back channels. I myself was _on_ IRC as word started leaking out, and still wasn't fully certain of what was going on or what to trust. Wichert's website (which I only learned was his the 27th!) was very helpful, as was the coverage provided by Slashdot and elsewhere. Discussion this with Manoj on IRC, my suggestion as summarized by him is that Debian should have an emergency response plan, part of which is a communications policy in the event a similar future compromise or systems failure. Specifically: - Triggering events. There are thresholds below which notifications needn't be triggered, and above which they very much should. Suggested: any event significantly affecting perceptions of security of the Debian archives or servers. Any outage of mail, web, or archive services anticipated to last beyond . E.g.: 6-12 hours, across core servers (but not mirrors). Any core server root compromise. *Not* single-package issues. Nuclear war or asteroid strike: you're on your own. - Where to provide information. Personal websites and news channels served well, but an advance statement of "here's where you should turn in the event of an emergency" would be useful. - What information to provide. Specifically, - the known (or unknown) status of archive or package compromise. - diagnostic checks; and/or - cleanup procedures. Wichert's pages on this would be a good template. By "known (or unkown)", I mean: if the archives are reasonably known to be safe, or are known to be compromised, this is communicated. If an assessment cannot be made with confidence, _that_ fact should be stated, e.g.: "the current security of the archives is unknown". By diagnostics and cleanup: pointers to tools or documentation explaining how to assess and/or secure a system. Wipe and rebuild if necessary. Again, wiggy.net
