Re: gluck (cvs, people, planet, etc.) downtime - ongoing raid problems

2005-04-05 Thread Christian Storch
Adam M. schrieb:
 Blars Blarson wrote:
 
 
Name Server:SAENS.DEBIAN.ORG
Name Server:KLECKER.DEBIAN.ORG
Name Server:SPOHR.DEBIAN.ORG
   


spohr changed IP addresses last week, and the glue record returned by
the .org nameservers still had the old address when I checked a few
hours ago.  This has been reported to debian-admin.  (The new address
is 140.211.166.43)

 

 
 
 Ok, but that should not cause DNS failure unless the old spohr address
 returned authoritative no such domain or the other two DNSes didn't work
 either.

You could be right, but when ORG-nameservers are returning NO glue record
as on last sunday night there is no chance.
It looks like these nameservers had lost their g-r's during update as
mentioned above. But why/how could this happen?

Christian


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]



gluck (cvs, people, planet, etc.) downtime - ongoing raid problems

2005-04-04 Thread James Troup
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

On Sunday evening gluck.debian.org started experiencing problems
writing to it's disks.  The local admins investigated and after
physically power cycling machine it became apparent that the RAID
controller was deeply unhappy - it claimed to have lost 2 out of it's
6 RAID5'd drives.  After reclaiming the drives it was left fscking
overnight for more than 9 hours.

Unfortunately, after it finished and came back up, it became apparent
that there's been extensive data corruption on the partition that
housed both /home and /org.  The kernel is also deeply unhappy with
the file system, despite it being declared clean by e2fsck, with both
ls and rsync managing to cause the kernel to oops.

We've taken the box down for now and are trying to recover what we can
and get the file system to a state where the kernel is at least on
speaking terms.  More information to come...

- -- 
James
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.0 (GNU/Linux)
Comment: Processed by Mailcrypt 3.5.8 http://mailcrypt.sourceforge.net/

iQIVAwUBQlGy9dfD8TGrKpH1AQJIIBAAjfaOcCcBLfDIDh1YlQ7T7/yRel06YGwj
FgGAnTwYQmHBEtSb2yRXWbJEMeyhYi5IvqkOVlFFY12lLrCs1jd5wqfX4WlST2pb
VNMqyRUQFYn5jkLP0Z8gLTShT5jqHarI4IJ0HQ3UbL6TOkrTfo/1PjTotSgOZalV
DLTAv7n/r7Mog21ZwbZHxR3Hdpj54DuNJ5AsicF0SpcmGIdkyRD+in+a9UehZ8mo
agdDv5iamGozncNCMpUhxK6dYNU3foexz/B06Lf721RArM/8pJF2MBeUzjPK1bBF
hv3rqHqRIETyGkr0L23VavtUovunP9OMFxBSdgQQMyYTcqDBK1ioP/RalAEQlKEl
W5ZUkBvZUbu3wh6HnFWGORo0ev4j9PfMKX6fX+uJNPYTz9t6RyJ07uVm8hdUJ8hZ
HydNEq+X87FukKGDGVE+phPCZc/tdoYYYnPRH6fh2DFvAqzmz1mHFhj2ypx8Qwq5
ksykX16p+4yBhTNDuxcSU8rPuc6uImo8Qgug09N9gDelHybwutIg/LpKdxVoPxNT
ADsId7yviCgdJOxZ4G5Ndo/dIN0RSo6TZh59gaQaRk2/CSOsX7C68ypaeIrWpA2U
H+U27dTZqP2ssJaT9pFNAF9439nOZbjnL9T7NVXcgCVgQchkAB+64JWa45nBUuLf
VboOE7/Ymp4=
=i8W6
-END PGP SIGNATURE-


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]



Re: gluck (cvs, people, planet, etc.) downtime - ongoing raid problems

2005-04-04 Thread Christian Storch
James Troup wrote:
 Hi,
 
 On Sunday evening gluck.debian.org started experiencing problems
 writing to it's disks.  The local admins investigated and after
 physically power cycling machine it became apparent that the RAID
 controller was deeply unhappy - it claimed to have lost 2 out of it's
 6 RAID5'd drives.  After reclaiming the drives it was left fscking
 overnight for more than 9 hours.
 

Strange: Could there be any correlation with my observed problems
about resolving anything of debian.org during exactly that time?
(http://lists.debian.org/debian-isp/2005/04/msg00023.html)

Christian


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]



Re: gluck (cvs, people, planet, etc.) downtime - ongoing raid problems

2005-04-04 Thread Adam M.
Christian Storch wrote:

Strange: Could there be any correlation with my observed problems
about resolving anything of debian.org during exactly that time?
(http://lists.debian.org/debian-isp/2005/04/msg00023.html)
  


Name Server:SAENS.DEBIAN.ORG
Name Server:KLECKER.DEBIAN.ORG
Name Server:SPOHR.DEBIAN.ORG

Gluck doesn't appear to host DNS of any kind.

- Adam



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]



Re: gluck (cvs, people, planet, etc.) downtime - ongoing raid problems

2005-04-04 Thread Blars Blarson
Name Server:SAENS.DEBIAN.ORG
Name Server:KLECKER.DEBIAN.ORG
Name Server:SPOHR.DEBIAN.ORG

spohr changed IP addresses last week, and the glue record returned by
the .org nameservers still had the old address when I checked a few
hours ago.  This has been reported to debian-admin.  (The new address
is 140.211.166.43)

-- 
Blars Blarson   [EMAIL PROTECTED]
http://www.blars.org/blars.html
With Microsoft, failure is not an option.  It is a standard feature.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]



Re: gluck (cvs, people, planet, etc.) downtime - ongoing raid problems

2005-04-04 Thread Adam M.
Blars Blarson wrote:

Name Server:SAENS.DEBIAN.ORG
Name Server:KLECKER.DEBIAN.ORG
Name Server:SPOHR.DEBIAN.ORG



spohr changed IP addresses last week, and the glue record returned by
the .org nameservers still had the old address when I checked a few
hours ago.  This has been reported to debian-admin.  (The new address
is 140.211.166.43)

  


Ok, but that should not cause DNS failure unless the old spohr address
returned authoritative no such domain or the other two DNSes didn't work
either.

- Adam



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]