I have an IBM System x 3250 that successfully boots using the standard
kernel (2.6.18-8.1.10.el5) but fails to boot with the xen kernel. The kernel
appears to fail when it attempts to initialize the driver for the onboard
SAS controller (LSI Logic / Symbios Logic SAS1064E PCI-Express Fusion-MPT
SAS). Can anyone provide any advice? Here's the relevant info. Please let me
know if any other information would be helpful.
IBM System x 3250: dual-core Pentium D 3GHz, 3GB memory
[EMAIL PROTECTED] ~]# lspci
00:00.0 Host bridge: Intel Corporation E7230/3000/3010 Memory Controller Hub
(rev c0)
00:01.0 PCI bridge: Intel Corporation E7230/3000/3010 PCI Express Root Port
(rev c0)
00:03.0 PCI bridge: Intel Corporation 82975X/3010 PCI Express Root Port (rev
c0)
00:1c.0 PCI bridge: Intel Corporation 82801G (ICH7 Family) PCI Express Port
1 (rev 01)
00:1c.4 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI
Express Port 5 (rev 01)
00:1c.5 PCI bridge: Intel Corporation 82801GR/GH/GHM (ICH7 Family) PCI
Express Port 6 (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #1
(rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #2
(rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #3
(rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI #4
(rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI
Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface
Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller
(rev 01)
00:1f.3 SMBus: Intel Corporation 82801G (ICH7 Family) SMBus Controller (rev
01)
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit
Ethernet PCI Express (rev 21)
03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit
Ethernet PCI Express (rev 21)
05:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1064E
PCI-Express Fusion-MPT SAS (rev 02)
1a:04.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02)
[EMAIL PROTECTED] ~]# rpm -qa | grep xen
kernel-xen-devel-2.6.18-8.1.14.el5
xen-libs-3.0.3-25.0.3.el5
kernel-xen-2.6.18-8.1.14.el5
xen-3.0.3-25.0.3.el5
xen-devel-3.0.3-25.0.3.el5
kernel output (via serial console):
Xen version 3.0.3-rc5-8.1.14.el5 (mockbuild@) (gcc version 4.1.1 20070105
(Red Hat 4.1.1-52)) Thu Sep 27 18:50:52 EDT 27
Latest ChangeSet:
unavailable
(XEN) Command line: /xen.gz-2.6.18-8.1.14.el5com1=38400,8n1
(XEN) Physical RAM
map:
(XEN) 0000000000000000 - 000000000009e800
(usable)
(XEN) 000000000009e800 - 00000000000a0000
(reserved)
(XEN) 00000000000d6000 - 00000000000d8000
(reserved)
(XEN) 00000000000e0000 - 0000000000100000
(reserved)
(XEN) 0000000000100000 - 00000000bfed0000
(usable)
(XEN) 00000000bfed0000 - 00000000bfee1000 (ACPI
data)
(XEN) 00000000bfee1000 - 00000000bff00000 (ACPI
NVS)
(XEN) 00000000bff00000 - 00000000c0000000
(reserved)
(XEN) 00000000fec00000 - 00000000fec10000
(reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ff000000 - 0000000100000000 (reserved)
(XEN) System RAM: 3070MB (3144120kB)
(XEN) Xen heap: 14MB (14356kB)
(XEN) found SMP MP-table at 000f68e0
(XEN) DMI present.
(XEN) Using APIC driver default
(XEN) ACPI: RSDP (v000 PTLTD ) @
0x00000000000f68b0
(XEN) ACPI: RSDT (v001 PTLTD RSDT 0x06040000 LTP 0x00000000) @
0x00000000bfed9b7a
(XEN) ACPI: FADT (v001 INTEL 0x06040000 PTL 0x00000003) @
0x00000000bfee0e70
(XEN) ACPI: MCFG (v001 PTLTD MCFG 0x06040000 LTP 0x00000000) @
0x00000000bfee0ee4
(XEN) ACPI: MADT (v001 PTLTD APIC 0x06040000 LTP 0x00000000) @
0x00000000bfee0f20
(XEN) ACPI: BOOT (v001 PTLTD $SBFTBL$ 0x06040000 LTP 0x00000001) @
0x00000000bfee0f88
(XEN) ACPI: SPCR (v001 PTLTD $UCRTBL$ 0x06040000 PTL 0x00000001) @
0x00000000bfee0fb0
(XEN) ACPI: SSDT (v001 PmRef CpuPm 0x00003000 INTL 0x20050228) @
0x00000000bfed9bb6
(XEN) ACPI: DSDT (v001 INTEL GLENWOOD 0x06040000 MSFT 0x0100000e) @
0x0000000000000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
(XEN) Processor #0 15:6 APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
(XEN) Processor #1 15:6 APIC version 20
(XEN) ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
(XEN) ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
(XEN) ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
(XEN) ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
(XEN) ACPI: IRQ0 used by override.
(XEN) ACPI: IRQ2 used by override.
(XEN) ACPI: IRQ9 used by override.
(XEN) Enabling APIC mode: Flat. Using 1 I/O APICs
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Initializing CPU#0
(XEN) Detected 3000.299 MHz processor.
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#0.
(XEN) CPU0: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU0: Thermal monitoring handled by SMI
(XEN) CPU0: Intel(R) Pentium(R) D CPU 3.00GHz stepping 05
(XEN) Booting processor 1/1 eip 90000
(XEN) Initializing CPU#1
(XEN) CPU: Trace cache: 12K uops, L1 D cache: 16K
(XEN) CPU: L2 cache: 2048K
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 1
(XEN) Intel machine check architecture supported.
(XEN) Intel machine check reporting enabled on CPU#1.
(XEN) CPU1: Intel P4/Xeon Extended MCE MSRs (24) available
(XEN) CPU1: Thermal monitoring enabled
(XEN) CPU1: Intel(R) Pentium(R) D CPU 3.00GHz stepping 05
(XEN) Total of 2 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using new ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) checking TSC synchronization across 2 CPUs: passed.
(XEN) Platform timer is 1.193MHz PIT
(XEN) Brought up 2 CPUs
(XEN) Machine check exception polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Domain 0 kernel supports features = { 0000001f }.
(XEN) Domain 0 kernel requires features = { 00000000 }.
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000002000000->0000000004000000 (731834 pages to be
allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80200000->ffffffff806cc804
(XEN) Init. ramdisk: ffffffff806cd000->ffffffff80c9c600
(XEN) Phys-Mach map: ffffffff80c9d000->ffffffff812425d0
(XEN) Start info: ffffffff81243000->ffffffff8124349c
(XEN) Page tables: ffffffff81244000->ffffffff81251000
(XEN) Boot stack: ffffffff81251000->ffffffff81252000
(XEN) TOTAL: ffffffff80000000->ffffffff81400000
(XEN) ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Initrd len 0x5cf600, start at 0xffffffff806cd000
(XEN) Scrubbing Free RAM: ...............................done.
(XEN) Xen trace buffers: disabled
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to
Xen).
kernel direct mapping tables up to b52ba000 @ 1254000-1db0000
Bootdata ok (command line is ro root=/dev/VolGroup01/LogVol00 noapic
acpi=off console=tty console=ttyS0 console=xvc xenc)
Linux version 2.6.18-8.1.14.el5xen ([EMAIL PROTECTED]) (gcc
version 4.1.1 20070105 (Red Hat 4.1.1-52)) #1 SM7
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 00000000b52ba000 (usable)
DMI present.
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: INTEL Product ID: GLENWOOD APIC at: 0xFEE00000
I/O APIC #2 Version 32 at 0xFEC00000.
Setting APIC routing to xen
Processors: 2
Allocating PCI resources starting at c4000000 (gap: c0000000:3ec00000)
Built 1 zonelists. Total pages: 742074
Kernel command line: ro root=/dev/VolGroup01/LogVol00 noapic acpi=off
console=tty console=ttyS0 console=xvc xenconsole=tc
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 32768 bytes)
Xen reported: 3000.298 MHz processor.
Console: colour VGA+ 80x25
Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
Software IO TLB enabled:
Aperture: 64 megabytes
Kernel range: 0xffff880004b83000 - 0xffff880008b83000
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Memory: 2823548k/2968296k available (2320k kernel code, 135900k reserved,
1312k data, 168k init)
Calibrating delay using timer specific routine.. 7504.29 BogoMIPS
(lpj=15008593)
Security Framework v1.0.0 initialized
SELinux: Initializing.
SELinux: Starting in permissive mode
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 256
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
CPU: Physical Processor ID: 0
CPU: Processor Core ID: 0
(SMP-)alternatives turned off
Brought up 2 CPUs
Initializing CPU#1
migration_cost=812
checking if image is initramfs... it is
Grant table initialized
NET: Registered protocol family 16
ACPI Exception (utmutex-0262): AE_BAD_PARAMETER, Thread 10387A0 could not
acquire Mutex [2] [20060707]
PCI: Using configuration type 1
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
xen_mem: Initialising balloon driver.
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: Probing PCI hardware
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.1
PCI: Transparent bridge - 0000:00:1e.0
PCI: Using IRQ router PIIX/ICH [8086/27b8] at 0000:00:1f.0
PCI: Found IRQ 11 for device 0000:00:1f.1
PCI: Sharing IRQ 11 with 0000:00:1d.2
PCI: Sharing IRQ 11 with 0000:1a:04.0
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
PCI: Bridge: 0000:00:01.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:03.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.0
IO window: 4000-4fff
MEM window: d0100000-d01fffff
PREFETCH window: c4000000-c40fffff
PCI: Bridge: 0000:00:1c.4
IO window: disabled.
MEM window: d0200000-d02fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1c.5
IO window: disabled.
MEM window: d0300000-d03fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: 5000-5fff
MEM window: d0400000-d04fffff
PREFETCH window: d8000000-dfffffff
PCI: Found IRQ 7 for device 0000:00:01.0
PCI: Sharing IRQ 7 with 0000:00:03.0
IRQ routing conflict for 0000:00:1c.0, have irq 10, want irq 7
IRQ routing conflict for 0000:00:1c.4, have irq 10, want irq 7
PCI: Sharing IRQ 7 with 0000:00:1d.3
PCI: Sharing IRQ 7 with 0000:05:00.0
PCI: Sharing IRQ 7 with 0000:01:00.0
PCI: Found IRQ 7 for device 0000:00:03.0
PCI: Sharing IRQ 7 with 0000:00:01.0
IRQ routing conflict for 0000:00:1c.0, have irq 10, want irq 7
IRQ routing conflict for 0000:00:1c.4, have irq 10, want irq 7
PCI: Sharing IRQ 7 with 0000:00:1d.3
PCI: Sharing IRQ 7 with 0000:05:00.0
PCI: Sharing IRQ 7 with 0000:01:00.0
PCI: Found IRQ 7 for device 0000:00:1c.0
PCI: Sharing IRQ 7 with 0000:00:01.0
PCI: Sharing IRQ 7 with 0000:00:03.0
IRQ routing conflict for 0000:00:1c.0, have irq 10, want irq 7
IRQ routing conflict for 0000:00:1c.4, have irq 10, want irq 7
PCI: Sharing IRQ 7 with 0000:00:1d.3
PCI: Sharing IRQ 7 with 0000:05:00.0
PCI: Sharing IRQ 7 with 0000:01:00.0
PCI: Found IRQ 7 for device 0000:00:1c.4
PCI: Sharing IRQ 7 with 0000:00:01.0
PCI: Sharing IRQ 7 with 0000:00:03.0
IRQ routing conflict for 0000:00:1c.0, have irq 10, want irq 7
IRQ routing conflict for 0000:00:1c.4, have irq 10, want irq 7
PCI: Sharing IRQ 7 with 0000:00:1d.3
PCI: Sharing IRQ 7 with 0000:05:00.0
PCI: Sharing IRQ 7 with 0000:01:00.0
PCI: Found IRQ 10 for device 0000:00:1c.5
IRQ routing conflict for 0000:00:1c.5, have irq 11, want irq 10
PCI: Sharing IRQ 10 with 0000:03:00.0
NET: Registered protocol family 2
IP route cache hash table entries: 131072 (order: 8, 1048576 bytes)
TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
IA-32 Microcode Update Driver: v1.14-xen <[EMAIL PROTECTED]>
audit: initializing netlink socket (disabled)
audit(1190996114.036:1): initialized
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
SELinux: Registering netfilter hooks
Initializing Cryptographic API
ksign: Installing public key data
Loading keyring
- Added public key AB7AA145C3566FE7
- User ID: CentOS (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
assign_interrupt_mode Found MSI capability
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Real Time Clock Driver v1.12ac
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
RAMDISK driver initialized: 16 RAM disks of 16384K size 4096 blocksize
Xen virtual console successfully installed as ttyS0
Event-channel device installed.
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH7: IDE controller at PCI slot 0000:00:1f.1
PCI: Found IRQ 11 for device 0000:00:1f.1
PCI: Sharing IRQ 11 with 0000:00:1d.2
PCI: Sharing IRQ 11 with 0000:1a:04.0
ICH7: chipset revision 1
ICH7: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0x30a0-0x30a7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0x30a8-0x30af, BIOS settings: hdc:pio, hdd:pio
hda: UJDA770 DVD/CDRW, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: No PS/2 controller found. Probing ports directly.
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
PCI: Sharing IRQ 7 with 0000:05:00.0
PCI: Sharing IRQ 7 with 0000:01:00.0
uhci_hcd 0000:00:1d.3: UHCI Host Controller
uhci_hcd 0000:00:1d.3: new USB bus registered, assigned bus number 4
uhci_hcd 0000:00:1d.3: irq 7, io base 0x00003060
usb usb4: configuration #1 chosen from 1 choice
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 2 ports detected
PCI: Found IRQ 5 for device 0000:00:1d.7
PCI: Sharing IRQ 5 with 0000:00:1d.0
ehci_hcd 0000:00:1d.7: EHCI Host Controller
ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 5
ehci_hcd 0000:00:1d.7: debug port 1
ehci_hcd 0000:00:1d.7: irq 5, io mem 0xd0000000
ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004
usb usb5: configuration #1 chosen from 1 choice
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 8 ports detected
SCSI subsystem initialized
Fusion MPT base driver 3.04.02
Copyright (c) 1999-2005 LSI Logic Corporation
Fusion MPT SAS Host driver 3.04.02
PCI: Found IRQ 7 for device 0000:05:00.0
PCI: Sharing IRQ 7 with 0000:00:01.0
PCI: Sharing IRQ 7 with 0000:00:03.0
IRQ routing conflict for 0000:00:1c.0, have irq 10, want irq 7
IRQ routing conflict for 0000:00:1c.4, have irq 10, want irq 7
PCI: Sharing IRQ 7 with 0000:00:1d.3
PCI: Sharing IRQ 7 with 0000:01:00.0
mptbase: Initiating ioc0 bringup
ioc0: SAS1064E: Capabilities={Initiator}
mptbase: Initiating ioc0 recovery
BUG: soft lockup detected on CPU#1!
Call Trace:
<IRQ> [<ffffffff802a76ad>] softlockup_tick+0xdb/0xed
[<ffffffff8026ba66>] timer_interrupt+0x396/0x3f2
[<ffffffff80210a87>] handle_IRQ_event+0x2d/0x60
[<ffffffff802a79ec>] __do_IRQ+0xa4/0x105
[<ffffffff802699b3>] do_IRQ+0xe7/0xf5
[<ffffffff8038dde8>] evtchn_do_upcall+0x86/0xe0
[<ffffffff8025cc1a>] do_hypervisor_callback+0x1e/0x2c
[<ffffffff8020c70a>] __delay+0x6/0x10
[<ffffffff880c2d0a>] :mptbase:WaitForDoorbellInt+0xbe/0xe9
[<ffffffff880c2f3f>] :mptbase:mpt_handshake_req_reply_wait+0x138/0x296
[<ffffffff880c38a3>] :mptbase:SendIocInit+0x242/0x325
[<ffffffff880c32d5>] :mptbase:GetIocFacts+0x7e/0x253
[<ffffffff880c4459>] :mptbase:MakeIocReady+0x62f/0xa67
[<ffffffff880c6b2b>] :mptbase:mpt_do_ioc_recovery+0xfbd/0xffd
[<ffffffff803282e8>] __next_cpu+0x19/0x28
[<ffffffff8027efa9>] find_busiest_group+0x1db/0x44a
[<ffffffff802809ca>] rebalance_tick+0x1be/0x411
[<ffffffff802958f4>] run_posix_cpu_timers+0x25/0x690
[<ffffffff880c6c1c>] :mptbase:mpt_HardResetHandler+0xb1/0x123
[<ffffffff880c6c8e>] :mptbase:mpt_timer_expired+0x0/0x24
[<ffffffff880c6c99>] :mptbase:mpt_timer_expired+0xb/0x24
[<ffffffff8028b7a4>] run_timer_softirq+0x13f/0x1c6
[<ffffffff80211ed3>] __do_softirq+0x62/0xdd
[<ffffffff8025d0e8>] call_softirq+0x1c/0x28
[<ffffffff80269b3d>] do_softirq+0x31/0x98
[<ffffffff802699b8>] do_IRQ+0xec/0xf5
[<ffffffff8038dde8>] evtchn_do_upcall+0x86/0xe0
[<ffffffff8025cc1a>] do_hypervisor_callback+0x1e/0x2c
<EOI> [<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
[<ffffffff802063aa>] hypercall_page+0x3aa/0x1000
[<ffffffff8026afe2>] raw_safe_halt+0x84/0xa8
[<ffffffff802684f8>] xen_idle+0x38/0x4a
[<ffffffff80247bcd>] cpu_idle+0x97/0xba
scsi0 : ioc0: LSISAS1064E, FwRev=01100000h, Ports=1, MaxQ=511, IRQ=7
mptbase: Initiating ioc0 recovery
BUG: soft lockup detected on CPU#1!
_______________________________________________
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
