[sorry for the dual send; formatting got hosed up the 1st time]
PROBLEM: The module usb-uhci consistantly causes Windows 2K
booting to fail when doing a warm reboot from Linux into
Windows 2000.
[1.] One line summary of the problem:
The module usb-uhci consistantly causes Windows 2K
booting to fail when doing a warm reboot from Linux into
Windows 2000.
[2.] Full description of the problem/report:
SHORT VERSION
The module usb-uhci consistantly causes Windows 2000
booting to fail when doing a warm reboot (aka:
shutdown -r, init 6, ctrl-alt-delete) from Linux into
Windows 2000 in my environment. During the Windows 2000 boot,
the system hangs.
When the usb-uhci module is removed (rmmod usb-uhci)
prior to the warm reboot, Windows boots fine.
KERNEL: I'm using the most recent stable kernel
(2.4.18) compiled under a fresh install of Red Hat 7.2
ALT DRIVER: I have tried the 'uhci' (alt) module
as well. The uhci seemed to exhibit the same behavior, but
I didn't do the same amount of testing so I'm not including
it in the problem description.
See the end of this doc for a workaround.
HUGELY LONG VERSION
(or, how I stopped worrying and learned to love the reboot =)
I have a Dell Inspiron 3500 that I dual-boot between
RedHat Linux and Windows 2000. I have some USB devices but I
have never used them with Linux.
Recently, I upgraded Red Hat from v6.2 to v7.2, by
formatting the partitions and reinstalling.
After doing this, I did a 'shutdown -r now' in order
to do a warm boot into Windows 2000. Unfortunately, Win2K
hung as it was booting. (Windows 2k generates no error
message as far as I can tell -- it just hangs). [See #6
below for more detail of how to recreate.]
Following this, I powered the box down entirely
(cold boot) and tried booting into Windows again. This time,
Windows booted fine.
I wouldn't have thought anything of this but I mentioned
it to a co-worker (who also has an older Dell laptop) and he
said that he was never able to reboot from Linux into Windows
using "shutdown -r". He said, however, that 'halt' or 'init 0'
would work fine. I tried it, and he was right - I was able
to boot just fine -- as long as it was a cold boot.
After some looking into what the technical difference
on the software side between 'halt' and 'reboot' I determined
that the problem must be related to doing a cold (vs. warm)
boot of the box.
Ok, so the next question I had was: What survives
a warm boot but not a cold boot? It's obviously not RAM
or disk (unless Linux is writing something to the boot
sector or Windows partion) so I guessed it must be related
to a hardware device.
I began by removing all optional devices from
the laptop (external floppy, PCMCIA modem/LAN cards), and
verifying that the warm reboot problem was still there. It
was.
Next, I tried commenting out all of the modules
in /etc/modules.conf. I was surprised to find that the
problem went away. Through the process of elimination,
I determined the line that was causing the problem was
"alias usb-controller usb-uhci".
If I commented the line out, did a cold boot
into Linux, and rebooted into Windows, W2K consistantly
booted successfully.
If I uncommented the line, did a cold boot
into Linux, and rebooted into Windows, W2K would consistantly
hang when booting.
My theory at this point is that it was related to
usbcore or usb-uhci. I tried booting without the alias
(meaning without the modules loaded), and manually doing a
'modprobe usbcore'. After the warm boot, things still worked
fine. This means 'usbcore' on its own was not the problem.
I went throught the procedure again. This time
I did a 'modprobe usbcore' followed by 'modprobe usb-uhci'.
Rebooted into Windows. It hung.
At this point, it seems like the usb-uhci module
is the problem. Here's my wild guess as to what's happening:
the usb-uhci is putting the USB controller in a state
that Win2K can't deal with. There are two ways to reset
the state of the controller:
1. Cold booting the machine.
2. Removing the usb-uhci module.
Maybe this has something to do with the device being assigned
an interrupt or I/O memory; at this point I'm just guessing
so I'll let the experts take it from here.
One other note: I've tried the 'uhci' (alt driver)
driver and reproduced the exact same problem.
I've included a workaround at the end of this
problem report that basically amounts to removing the
usb-uhci module prior to a reboot.
I have no idea how hardware specific this is, but
I'd be curious to know if anyone else can reproduce it or
if there is any better way to fix this.
[3.] Keywords (i.e., modules, networking, kernel):
shutdown -r, halt, reboot, warm reboot
Dell Inspiron 3500 laptop, i686, Pentium II
Intel 82371AB USB Controller, UHCI,
usb-uhci module, usbcore module
Red Hat 7.2, 2.4.7 kernel, 2.4.18 kernel, USB
Windows 2000 freeze, hang, stall
[4.] Kernel version (from /proc/version):
I've reproduced it in the following kernels:
2.4.7-10 - The default that ships with RedHat 7.2
2.4.18 - Latest stable kernel, downloaded from kernel.org
and compiled using config file:
/usr/src/linux-2.4.7-10/configs/kernel-2.4.7-i686.config
(I tweaked it to set CONFIG_USB_DEBUG=y)
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
N/A
[6.] A small shell script or example program which triggers the
problem (if possible)
The problem happens in the default Red Hat 7.2 release.
Recreating the problem
----------------------
This may be specific to my environment (machine, USB
controller, etc.) but here it is:
1. Do a clean install of Red Hat 7.2 (kernel=2.4.7-10 or 2.4.18)
onto a system that dual-boots into Windows 2000.
2. After the installation is complete, boot into Linux.
3. As root, run:
# /sbin/shutdown -r now
(Note: You can also run '/sbin/init 6' or '/sbin/reboot',
but '/sbin/halt' or '/sbin/init 0' will *NOT* cause
the problem.)
4. After rebooting, select to boot Windows 2K from the LILO
prompt.
During the Windows reboot process, the following happens:
* Windows 2000 does the black "Staring WIndows" progress meter,
which completes successfully.
* Windows 2000 brings up the white "MS Windows 2000 Professional"
screen, with the "Staring up..." progress meter. The progress
meter gets a bit past 50%. At that point the moving
animation (just above the progress meter) stops,
and Windows hangs.
How I determined that it was related to the USB driver
------------------------------------------------------
1. Boot into Red Hat Linux.
2. Run the following commands as root:
# /sbin/rmmod usbcore
# /sbin/rmmod usb-uhci
# /sbin/shutdown -r now
3. When LILO comes up, boot into Windows 2000.
4. Windows 2000 now boots fine (it works).
Also, see the workaround at the end of this document.
[7.] Environment
Hardware:
Dell Inspiron 3500 laptop
Intel 82371AB/EB USB Controller (UHCI)
No USB devices connected.
PCMCIA cards:
3COM network card.
3COM combo network/modem card
external floppy drive
(Note: problem has been reproduced both with
and without PCMCIA cards & floppy drive)
Operating Systems (dual-boot using LILO):
Red Hat Linux 7.2
- using default kernel (2.4.7-10)
- using latest stable kernel and modules (2.4.18)
Microsoft Windows 2000 (5.00.2195, service pack 2)
[7.1.] Software (add the output of the ver_linux script here)
# sh scripts/ver_linux
If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.
Linux localhost.localdomain 2.4.18 #1 Thu Apr 25 17:41:58 CDT 2002 i686
unknown
Gnu C 2.96
Gnu make 3.79.1
binutils 2.11.90.0.8
util-linux 2.11f
mount 2.11g
modutils 2.4.6
e2fsprogs 1.23
reiserfsprogs 3.x.0j
pcmcia-cs 3.1.22
PPP 2.4.1
isdn4k-utils 3.1pre1
Linux C Library 2.2.4
Dynamic linker (ldd) 2.2.4
Procps 2.0.7
Net-tools 1.60
Console-tools 0.3.3
Sh-utils 2.0.11
Modules Loaded smbfs ide-cd cdrom ad1848 sound soundcore
binfmt_misc serial_cs 3c574_cs ds yenta_socket
pcmcia_core autofs appletalk ipx ipchains ext3
jbd usb-uhci usbcore
[7.2.] Processor information (from /proc/cpuinfo):
# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 6
model name : Mobile Pentium II
stepping : 10
cpu MHz : 330.879
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep
mtrr pge mca cmov pat pse36 mmx fxsr
bogomips : 660.27
[7.3.] Module information (from /proc/modules):
# cat /proc/modules
smbfs 34704 1 (autoclean)
ide-cd 26816 0 (autoclean)
cdrom 27840 0 (autoclean) [ide-cd]
ad1848 21584 0 (autoclean) (unused)
sound 56896 0 (autoclean) [ad1848]
soundcore 4048 2 (autoclean) [sound]
binfmt_misc 6128 1
serial_cs 4416 0 (unused)
3c574_cs 8992 1
ds 6960 2 [serial_cs 3c574_cs]
yenta_socket 9104 2
pcmcia_core 41216 0 [serial_cs 3c574_cs ds yenta_socket]
autofs 10368 0 (autoclean) (unused)
appletalk 20720 0 (autoclean)
ipx 16160 0 (autoclean)
ipchains 35936 13
ext3 61360 1 (autoclean)
jbd 40016 1 (autoclean) [ext3]
usb-uhci 21664 0 (unused)
usbcore 55680 1 [usb-uhci]
[7.4.] Loaded driver and hardware information (/proc/ioports, /proc/iomem)
Note: usb-uhci seems to be grabbing interrupt 11, io: fce0-fcff
# cat /proc/ioports
0000-001f : dma1
0020-003f : pic1
0040-005f : timer
0060-006f : keyboard
0070-007f : rtc
0080-008f : dma page reg
00a0-00bf : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
02f8-02ff : serial(auto)
0320-033f : 3c574_cs
0376-0376 : ide1
03c0-03df : vga+
03f6-03f6 : ide0
03f8-03ff : serial(set)
0cf8-0cff : PCI conf1
2180-219f : Intel Corp. 82371AB PIIX4 ACPI
4000-40ff : PCI CardBus #02
4400-44ff : PCI CardBus #02
4800-48ff : PCI CardBus #06
4c00-4cff : PCI CardBus #06
8000-803f : Intel Corp. 82371AB PIIX4 ACPI
fcd0-fcdf : Intel Corp. 82371AB PIIX4 IDE
fcd0-fcd7 : ide0
fcd8-fcdf : ide1
fce0-fcff : Intel Corp. 82371AB PIIX4 USB
fce0-fcff : usb-uhci
# cat /proc/iomem
00000000-0009f7ff : System RAM
0009f800-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000f0000-000fffff : System ROM
00100000-07feffff : System RAM
00100000-00208a6f : Kernel code
00208a70-0024fe8b : Kernel data
07ff0000-07fffbff : ACPI Tables
07fffc00-07ffffff : ACPI Non-volatile Storage
10000000-10000fff : Texas Instruments PCI1220
10001000-10001fff : Texas Instruments PCI1220 (#2)
10400000-107fffff : PCI CardBus #02
10800000-10bfffff : PCI CardBus #02
10c00000-10ffffff : PCI CardBus #06
11000000-113fffff : PCI CardBus #06
a0000000-a0000fff : card services
a0001000-a0001fff : card services
e0000000-e3ffffff : Intel Corp. 440BX/ZX - 82443BX/ZX Host bridge
fd000000-fe3fffff : PCI Bus #01
fd000000-fdffffff : Neomagic Corporation [MagicMedia 256AV]
fe000000-fe3fffff : Neomagic Corporation [MagicMedia 256AV Audio]
fe700000-fecfffff : PCI Bus #01
fe700000-fe7fffff : Neomagic Corporation [MagicMedia 256AV Audio]
fe800000-febfffff : Neomagic Corporation [MagicMedia 256AV]
fec00000-fecfffff : Neomagic Corporation [MagicMedia 256AV]
fffeb000-ffffffff : reserved
Windows 2000 IO Memory(from Device Manager):
I'm only showing the USB controller; let me know if you need more:
[0000FCE0 - 0000FCFF] Intel 82371 AB/EB PCI to USB Universal Host
Controller
Windows 2000 Interrupts (from Device Manager):
(ISA) 0 System Timer
(ISA) 1 PC/AT Enhanced PS/2 Keyboard (101/102-Key)
(ISA) 3 Communications Port (COM1)
(ISA) 4 3Com Megahertz (B) 10-100 LAN + 56K Modem (Ethernet)
(ISA) 4 3Com Megahertz (B) 10-100 LAN + 56K Modem PC Card (Modem
Interface)
(ISA) 5 NeoMagic MagicMedia 256AV NMA2 Codec (WDM)
(ISA) 6 Standard floppy disk controller
(ISA) 7 3Com-U.S. Robotics 3056
(ISA) 8 System CMOS/real time clock
(ISA) 12 Microsoft PS/2 Port Mouse (IntelliPoint)
(ISA) 13 Numeric data processor
(ISA) 14 Primary IDE Channel
(ISA) 15 Secondary IDE Channel
(PCI) 10 Texas Instruments PCI-1220 CardBus Controller
(PCI) 10 Texas Instruments PCI-1220 CardBus Controller
(PCI) 11 Intel 82371AB/EB PCI to USB Universal Host Controller
(PCI) 11 NeoMagic MagicMedia 256AV Audio Driver(WDM)
[7.5.] PCI information ('lspci -vvv' as root)
00:00.0 Host bridge: Intel Corporation 440BX/ZX - 82443BX/ZX Host bridge
(rev 03)
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Latency: 64
Region 0: Memory at e0000000 (32-bit, prefetchable) [size=64M]
Capabilities: [a0] AGP version 1.0
Status: RQ=31 SBA+ 64bit- FW- Rate=x1
Command: RQ=0 SBA- AGP- 64bit- FW- Rate=<none>
00:01.0 PCI bridge: Intel Corporation 440BX/ZX - 82443BX/ZX AGP bridge
(rev 03) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 128
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
I/O behind bridge: 0000f000-00000fff
Memory behind bridge: fe700000-fecfffff
Prefetchable memory behind bridge: fd000000-fe3fffff
BridgeCtl: Parity- SERR- NoISA+ VGA+ MAbort- >Reset- FastB2B+
00:04.0 CardBus bridge: Texas Instruments PCI1220 (rev 02)
Subsystem: Dell Computer Corporation: Unknown device 008f
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 168, cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at 10000000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=02, subordinate=05, sec-latency=176
Memory window 0: 10400000-107ff000 (prefetchable)
Memory window 1: 10800000-10bff000
I/O window 0: 00004000-000040ff
I/O window 1: 00004400-000044ff
BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset+ 16bInt+
PostWrite+
16-bit legacy interface ports at 0001
00:04.1 CardBus bridge: Texas Instruments PCI1220 (rev 02)
Subsystem: Dell Computer Corporation: Unknown device 008f
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 168, cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at 10001000 (32-bit, non-prefetchable) [size=4K]
Bus: primary=00, secondary=06, subordinate=09, sec-latency=176
Memory window 0: 10c00000-10fff000 (prefetchable)
Memory window 1: 11000000-113ff000
I/O window 0: 00004800-000048ff
I/O window 1: 00004c00-00004cff
BridgeCtl: Parity- SERR- ISA- VGA- MAbort- >Reset+ 16bInt+
PostWrite+
16-bit legacy interface ports at 0001
00:07.0 Bridge: Intel Corporation 82371AB PIIX4 ISA (rev 02)
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:07.1 IDE interface: Intel Corporation 82371AB PIIX4 IDE (rev 01)
(prog-if 80 [Master])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Region 4: I/O ports at fcd0 [size=16]
00:07.2 USB Controller: Intel Corporation 82371AB PIIX4 USB (rev 01)
(prog-if 00 [UHCI])
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 64
Interrupt: pin D routed to IRQ 11
Region 4: I/O ports at fce0 [size=32]
00:07.3 Bridge: Intel Corporation 82371AB PIIX4 ACPI (rev 02)
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin ? routed to IRQ 9
01:00.0 VGA compatible controller: Neomagic Corporation [MagicMedia
256AV] (rev 12) (prog-if 00 [VGA])
Subsystem: Dell Computer Corporation: Unknown device 008f
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B+
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 128 (4000ns min, 63750ns max)
Interrupt: pin A routed to IRQ 0
Region 0: Memory at fd000000 (32-bit, prefetchable) [size=16M]
Region 1: Memory at fe800000 (32-bit, non-prefetchable) [size=4M]
Region 2: Memory at fec00000 (32-bit, non-prefetchable) [size=1M]
Capabilities: [dc] Power Management version 1
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
01:00.1 Multimedia audio controller: Neomagic Corporation [MagicMedia
256AV Audio] (rev 12)
Subsystem: Dell Computer Corporation MagicMedia 256AV Audio
Device on Colorado Inspiron
Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B+
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin B routed to IRQ 11
Region 0: Memory at fe000000 (32-bit, prefetchable) [size=4M]
Region 1: Memory at fe700000 (32-bit, non-prefetchable) [size=1M]
Capabilities: [dc] Power Management version 1
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
[7.6.] SCSI information (from /proc/scsi/scsi)
Probably N/A; let me know if you think you need this.
[7.7.] Other information that might be relevant to the problem
(please look in /proc and include all information that you
think to be relevant):
USB devices
==================================================
Note: I do not have any USB devices attached to the machine during this
problem.
# cat /proc/bus/usb/devices
T: Bus=01 Lev=00 Prnt=00 Port=00 Cnt=00 Dev#= 1 Spd=12 MxCh= 2
B: Alloc= 0/900 us ( 0%), #Int= 0, #Iso= 0
D: Ver= 1.00 Cls=09(hub ) Sub=00 Prot=00 MxPS= 8 #Cfgs= 1
P: Vendor=0000 ProdID=0000 Rev= 0.00
S: Product=USB UHCI Root Hub
S: SerialNumber=fce0
C:* #Ifs= 1 Cfg#= 1 Atr=40 MxPwr= 0mA
I: If#= 0 Alt= 0 #EPs= 1 Cls=09(hub ) Sub=00 Prot=00 Driver=hub
E: Ad=81(I) Atr=03(Int.) MxPS= 8 Ivl=255ms
Lines generated by USB debugging setting in kernel.
==================================================
> usb.c: registered new driver usbdevfs
> usb.c: registered new driver hub
> usb-uhci.c: $Revision: 1.275 $ time 18:57:58 Apr 25 2002
> usb-uhci.c: High bandwidth mode enabled
> PCI: Found IRQ 11 for device 00:07.2
> usb-uhci.c: USB UHCI at I/O 0xfce0, IRQ 11
> usb-uhci.c: Detected 2 ports
> usb.c: new USB bus registered, assigned bus number 1
> usb.c: kmalloc IF c2e00460, numif 1
> usb.c: new device strings: Mfr=0, Product=2, SerialNumber=1
> usb.c: USB device number 1 default language ID 0x0
> Product: USB UHCI Root Hub
> SerialNumber: fce0
> hub.c: USB hub found
> hub.c: 2 ports detected
> hub.c: standalone hub
> hub.c: ganged power switching
> hub.c: global over-current protection
> hub.c: Port indicators are not supported
> hub.c: power on to power good time: 2ms
> hub.c: hub controller current requirement: 0mA
> hub.c: port removable status: RR
> hub.c: local power source is good
> hub.c: no over-current condition exists
> hub.c: enabling power on all ports
> usb.c: hub driver claimed interface c2e00460
> usb.c: kusbd: /sbin/hotplug add 1
> usb-uhci.c: v1.275:USB Universal Host Controller Interface driver
[X.] Other notes, patches, fixes, workarounds:
WORKAROUND
As a workaround, I altered the halt script to automatically
remove the usbcore/usb-uhci before a warm reboot
(meaning, '/sbin/shutdown -r now', '/sbin/init 6', '/sbin/reboot',
or ctrl-alt-delete).
Add the following lines to /etc/rc.d/init.d/halt before the
last line in the file (eval $command $HALTARGS):
# Remove USB modules to avoid warm boot problem that
# causes Windows 2000 not to boot properly.
if [ "$command" = reboot ] ; then
echo $"Removing USB modules to avoid Win2K reboot
problem."
rmmod usb-uhci
rmmod usbcore
fi
__________________________________________________
Do You Yahoo!?
Yahoo! Games - play chess, backgammon, pool and more
http://games.yahoo.com/
_______________________________________________
[EMAIL PROTECTED]
To unsubscribe, use the last form field at:
https://lists.sourceforge.net/lists/listinfo/linux-usb-users
