Re: r8169 ethernet bonding problems
Tim Durack [EMAIL PROTECTED] : [snip] Can you try 2.6.22-rc5 + http://www.fr.zoreil.com/people/francois/misc/20070619-2.6.22-rc5-r8169-test.patch It does not seem to work too bad here (tested with 2.6.22-rc5 + patch): # grep bonding /etc/modprobe.conf alias bond0 bonding options bonding miimon=1000 mode=balance-rr $ dmesg [0.00] Linux version 2.6.22-rc5 ([EMAIL PROTECTED]) (gcc version 4.1.1 20070105 (Red Hat 4.1.1-51)) #9 Tue Jun 19 00:19:24 CEST 2007 [...] [ 35.516210] r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded [ 35.516251] ACPI: PCI Interrupt :01:00.0[A] - GSI 17 (level, low) - IRQ 17 [ 35.516279] PCI: Setting latency timer of device :01:00.0 to 64 [ 35.516632] eth0: RTL8168b/8111b at 0xf8a08000, 00:13:8f:ea:b1:5d, XID 3800 IRQ 17 [ 35.523173] r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded [ 35.523204] ACPI: PCI Interrupt :03:01.0[A] - GSI 22 (level, low) - IRQ 20 [ 35.523473] eth1: RTL8110s at 0xf8a0a800, 00:09:5b:bd:c1:a5, XID 0400 IRQ 20 [ 35.526654] r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded [ 35.526683] ACPI: PCI Interrupt :03:02.0[A] - GSI 23 (level, low) - IRQ 21 [ 35.526940] eth2: RTL8110s at 0xf8a0cc00, 00:09:5b:bd:c1:a5, XID 0400 IRQ 21 [ 35.581378] USB Universal Host Controller Interface driver v3.0 [ 35.582010] ACPI: PCI Interrupt :00:1d.0[A] - GSI 23 (level, low) - IRQ 21 [ 35.582028] PCI: Setting latency timer of device :00:1d.0 to 64 [...] [ 46.300540] Ethernet Channel Bonding Driver: v3.1.2 (January 20, 2007) [ 46.307134] bonding: MII link monitoring set to 1000 ms [ 50.515819] r8169: eth1: link up [ 50.519218] r8169: eth1: link up [ 50.539460] bonding: bond0: enslaving eth1 as an active interface with an up link. [ 50.656913] r8169: eth2: link up [ 50.660296] r8169: eth2: link up [ 50.679294] bonding: bond0: enslaving eth2 as an active interface with an up link. $ ifconfig bond0 Link encap:Ethernet HWaddr 00:09:5B:BD:C1:A5 inet addr:192.168.1.3 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::209:5bff:febd:c1a5/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:319 errors:0 dropped:0 overruns:0 frame:0 TX packets:329 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:31583 (30.8 KiB) TX bytes:30884 (30.1 KiB) eth1 Link encap:Ethernet HWaddr 00:09:5B:BD:C1:A5 inet6 addr: fe80::209:5bff:febd:c1a5/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:158 errors:0 dropped:0 overruns:0 frame:0 TX packets:165 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:16228 (15.8 KiB) TX bytes:15418 (15.0 KiB) Interrupt:20 Base address:0xa800 eth2 Link encap:Ethernet HWaddr 00:09:5B:BD:C1:A5 inet6 addr: fe80::209:5bff:febd:c1a5/64 Scope:Link UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:161 errors:0 dropped:0 overruns:0 frame:0 TX packets:164 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:15355 (14.9 KiB) TX bytes:15466 (15.1 KiB) Interrupt:21 Base address:0xcc00 loLink encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:560 (560.0 b) TX bytes:560 (560.0 b) # ethtool -S eth1 NIC statistics: tx_packets: 205 rx_packets: 195 tx_errors: 0 rx_errors: 0 rx_missed: 0 align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 unicast: 192 broadcast: 1 multicast: 2 tx_aborted: 0 tx_underrun: 0 # ethtool -S eth2 NIC statistics: tx_packets: 205 rx_packets: 198 tx_errors: 0 rx_errors: 0 rx_missed: 0 align_errors: 0 tx_single_collisions: 0 tx_multi_collisions: 0 unicast: 195 broadcast: 2 multicast: 1 tx_aborted: 0 tx_underrun: 0 Let's play gently with the cables: PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.276 ms 64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.184 ms 64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.195 ms 64 bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=0.173 ms 64 bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=0.212 ms 64 bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=0.173 ms 64 bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=0.198 ms 64 bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=0.173 ms 64 bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=0.200 ms 64 bytes from 192.168.1.1: icmp_seq=10
Re: r8169 ethernet bonding problems
Yup. Works fine. The randomly changing interface order is proving to be a challenge! udev should be your friend to do a BUS/ID based rename. True, but so far I have to force link addrs, force promisc mode, and now add a udev work-around ;-| Strange thing is, when the interfaces get initialised as eth0/1/2 in order, they all have the correct macs. When they appear out of order (eth5/1/2, eth5/6/1 etc), at least one or more of the macs are messed up. Once yesterday I saw them initialise in order, with correct macs. Have not been able to repeat or figure out if there is some way of triggering this correct behaviour. Any more driver patches I can beta test? Tim: - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
On 5/2/07, Tim Durack [EMAIL PROTECTED] wrote:
Yup. Works fine. The randomly changing interface order is proving to
be a challenge!
udev should be your friend to do a BUS/ID based rename.
Okay, the udev workaround isn't too painful.
Would be great if I could do something like
ATTRS{address}=00:30:18:ab:cd:ef and have that force the mac during
initialisation.
Tim:
-
To unsubscribe from this list: send the line unsubscribe netdev in
the body of a message to [EMAIL PROTECTED]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3) after link failure (i.e. failover does not work) would be welcome. Okay, after fixing STP forwarding delays on the switch, bonding failover now works, as long as member links are forced to promisc mode. Still Have a problem with links randomly failing to achieve link on boot. This then requires a modprobe -r r8169; modprobe r8169 cycle to get link. Side effect is the mac address corruption I have described previously. Tim: - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Tim Durack [EMAIL PROTECTED] : [...] Still Have a problem with links randomly failing to achieve link on boot. This then requires a modprobe -r r8169; modprobe r8169 cycle to get link. Side effect is the mac address corruption I have described previously. Can you try to set the MAC address by hand with the completely patched kernel (i.e. including the mac-address-change patch) ? Given the last report in http://bugzilla.kernel.org/show_bug.cgi?id=6032, it should be straightforward with your Debian system. No clear idea why your three cards are not handled the same though. -- Ueimor Anybody got a battery for my Ultra 10 ? - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Can you try to set the MAC address by hand with the completely patched kernel (i.e. including the mac-address-change patch) ? Yup. Works fine. The randomly changing interface order is proving to be a challenge! Given the last report in http://bugzilla.kernel.org/show_bug.cgi?id=6032, it should be straightforward with your Debian system. No clear idea why your three cards are not handled the same though. - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Tim Durack [EMAIL PROTECTED] : Can you try to set the MAC address by hand with the completely patched kernel (i.e. including the mac-address-change patch) ? Yup. Works fine. The randomly changing interface order is proving to be a challenge! udev should be your friend to do a BUS/ID based rename. -- Ueimor Anybody got a battery for my Ultra 10 ? - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Can you revert patch #0012 and see if it changes this part of the problem ? Reverted. mac is still wrong - I assume the eprom got corrupted somehow. ~# modprobe r8169 r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded ACPI: PCI Interrupt :00:09.0[A] - GSI 18 (level, low) - IRQ 16 eth1: RTL8169sc/8110sc at 0xf8838000, 00:00:00:00:68:93, IRQ 16 r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded ACPI: PCI Interrupt :00:0b.0[A] - GSI 19 (level, low) - IRQ 17 eth2: RTL8169sc/8110sc at 0xf883a000, 00:30:18:ab:68:94, IRQ 17 r8169 Gigabit Ethernet driver 2.2LK-NAPI loaded ACPI: PCI Interrupt :00:0c.0[A] - GSI 16 (level, low) - IRQ 18 eth4: RTL8169sc/8110sc at 0xf8a76000, 00:30:18:ab:68:95, IRQ 18 I can work around this with an ip links set dev eth5 addr for now. Strange things is, this group of interfaces used to get initialised as eth0/1/2, in sequence. Now one of the interfaces is always out of order. A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3) after link failure (i.e. failover does not work) would be welcome. Will get these shortly. lspci -vx: 00:00.0 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge Subsystem: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge Flags: bus master, 66MHz, medium devsel, latency 8 Memory at e800 (32-bit, prefetchable) [size=128M] Capabilities: [80] AGP version 3.5 Capabilities: [50] Power Management version 2 00: 06 11 14 03 06 00 30 22 00 00 00 06 00 08 80 00 10: 08 00 00 e8 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 06 11 14 03 30: 00 00 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00:00.1 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge Flags: bus master, medium devsel, latency 0 00: 06 11 14 13 06 00 00 02 00 00 00 06 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:00.2 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge Flags: bus master, medium devsel, latency 0 00: 06 11 14 23 06 00 00 02 00 00 00 06 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:00.3 Host bridge: VIA Technologies, Inc. PT890 Host Bridge Flags: bus master, medium devsel, latency 0 00: 06 11 08 32 06 00 00 02 00 00 00 06 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:00.4 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge Flags: bus master, medium devsel, latency 0 00: 06 11 14 43 06 00 00 02 00 00 00 06 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:00.7 Host bridge: VIA Technologies, Inc. CN700/VN800/P4M800CE/Pro Host Bridge Flags: bus master, medium devsel, latency 0 00: 06 11 14 73 06 00 00 02 00 00 00 06 00 00 00 00 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 a0 00 00 00 00 00 00 00 00 00 00 00 00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI Bridge (prog-if 00 [Normal decode]) Flags: bus master, 66MHz, medium devsel, latency 0 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: d000-dfff Memory behind bridge: fb00-fcff Prefetchable memory behind bridge: f400-f7ff Capabilities: [70] Power Management version 2 00: 06 11 98 b1 07 00 30 02 00 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 00 01 01 00 d0 d0 20 e2 20: 00 fb f0 fc 00 f4 f0 f7 00 00 00 00 00 00 00 00 30: 00 00 00 00 70 00 00 00 00 00 00 00 00 00 08 00 00:09.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10) Subsystem: Jetway Information Co., Ltd. Unknown device 10ec Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 16 I/O ports at f200 [size=256] Memory at fdfff000 (32-bit, non-prefetchable) [size=256] [virtual] Expansion ROM at 4000 [disabled] [size=64K] Capabilities: [dc] Power Management version 2 00: ec 10 69 81 17 00 b0 02 10 00 00 02 08 40 00 00 10: 01 f2 00 00 00 f0 ff fd 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 f3 16 ec 10 30: 00 00 00 00 dc 00 00 00 00 00 00 00 0b 01 20 40 00:0a.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 80) (prog-if 10 [OHCI]) Subsystem: VIA Technologies, Inc. IEEE 1394 Host Controller Flags: bus master, stepping, medium devsel, latency 32, IRQ 17 Memory at fdffe000 (32-bit, non-prefetchable) [size=2K]
Re: r8169 ethernet bonding problems
A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3) after link failure (i.e. failover does not work) would be welcome. See attached. Captures performed on bond0. Let me know if you prefer captures against the physical member interfaces. Tim: promisc-off.pcap Description: Binary data promisc-on.pcap Description: Binary data fail-eth1.pcap Description: Binary data recover-eth1-1.pcap Description: Binary data fail-eth2-1.pcap Description: Binary data recover-eth2.pcap Description: Binary data
Re: r8169 ethernet bonding problems
On 4/30/07, Tim Durack [EMAIL PROTECTED] wrote: A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3) after link failure (i.e. failover does not work) would be welcome. See attached. Captures performed on bond0. Let me know if you prefer captures against the physical member interfaces. Hmm, see attached files with content this time. promisc-off.pcap Description: Binary data promisc-on.pcap Description: Binary data fail-eth1.pcap Description: Binary data recover-eth1.pcap Description: Binary data fail-eth2.pcap Description: Binary data recover-eth2.pcap Description: Binary data
Re: r8169 ethernet bonding problems
See: http://www.fr.zoreil.com/linux/kernel/2.6.x/2.6.21-rc7/ Please Cc: netdev on success/failure. Applied suggested patchset to vanilla 2.6.21. Compiled using .config from linux-image-2.6.21-rc7 pulled down from http://kernel-archive.buildserver.net/debian-kernel Compiles, boots. Bonding does not work reliably. Simple config: #auto bond0 iface bond0 inet manual up ip link set dev $IFACE up up echo +eth1 /sys/class/net/$IFACE/bonding/slaves up echo +eth2 /sys/class/net/$IFACE/bonding/slaves up ip link set dev eth1 up up ip link set dev eth2 up up ip addr add 172.21.255.5/29 dev $IFACE down ip addr del 172.21.255.5/29 dev $IFACE down ip link set dev eth2 down down ip link set dev eth1 down down echo -eth2 /sys/class/net/$IFACE/bonding/slaves down echo -eth1 /sys/class/net/$IFACE/bonding/slaves down ip link set dev $IFACE down Bond comes up, cannot ping unless I manually force promisc on both member links: ip link set dev eth1 promisc on ip link set dev eth2 promisc off Link failover does not work reliably. Carrier-detect appears to be working, so I don't think that is an issue. A disturbing problem is mac corruption after modprobe -r r8169; modprobe r8169 ip link sh: 2: eth5: BROADCAST,MULTICAST mtu 1500 qdisc noop qlen 1000 link/ether 00:00:00:00:68:93 brd ff:ff:ff:ff:ff:ff 3: eth1: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:30:18:ab:68:94 brd ff:ff:ff:ff:ff:ff 4: eth2: BROADCAST,MULTICAST mtu 1500 qdisc noop qlen 1000 link/ether 00:30:18:ab:68:95 brd ff:ff:ff:ff:ff:ff eth5 mac was 00:30:18:ab:68:93 before the modprobe. Not sure what's going on here, but I'm starting to think these interfaces aren't the best in the world... Tim: - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Tim Durack [EMAIL PROTECTED] wrote: [...] Bond comes up, cannot ping unless I manually force promisc on both member links: ip link set dev eth1 promisc on ip link set dev eth2 promisc off From looking at the source code for r8169, but it appears that the r8169 driver only reads the device MAC address at probe time, and doesn't propogate changes to the MAC out to the device. That might explain the promisc problem if the device also does MAC filtering. Am I missing something in the driver? I don't actually have one of these to test with, I'm just looking at the source and observing that dev_addr is only ever referenced in the probe function, and that's only to read in the MAC from the device. -J --- -Jay Vosburgh, IBM Linux Technology Center, [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
ip link set dev eth1 promisc on ip link set dev eth2 promisc off typo, ip link set dev eth2 promisc on of course From looking at the source code for r8169, but it appears that the r8169 driver only reads the device MAC address at probe time, and doesn't propogate changes to the MAC out to the device. That might explain the promisc problem if the device also does MAC filtering. Am I missing something in the driver? I have applied all patches from the set, including: 0012-r8169-mac-address-change-support.patch which I assume adds the ability to reprogram MAC filtering. Behaviour indicates the MAC filter doesn't get reprogrammed though. I don't actually have one of these to test with, I'm just looking at the source and observing that dev_addr is only ever referenced in the probe function, and that's only to read in the MAC from the device. You're not missing much, believe me :-| Tim: - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Tim Durack [EMAIL PROTECTED] : [...] Link failover does not work reliably. Carrier-detect appears to be working, so I don't think that is an issue. A disturbing problem is mac corruption after modprobe -r r8169; modprobe r8169 ip link sh: 2: eth5: BROADCAST,MULTICAST mtu 1500 qdisc noop qlen 1000 link/ether 00:00:00:00:68:93 brd ff:ff:ff:ff:ff:ff 3: eth1: BROADCAST,MULTICAST mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:30:18:ab:68:94 brd ff:ff:ff:ff:ff:ff 4: eth2: BROADCAST,MULTICAST mtu 1500 qdisc noop qlen 1000 link/ether 00:30:18:ab:68:95 brd ff:ff:ff:ff:ff:ff eth5 mac was 00:30:18:ab:68:93 before the modprobe. Can you revert patch #0012 and see if it changes this part of the problem ? A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3) after link failure (i.e. failover does not work) would be welcome. Not sure what's going on here, but I'm starting to think these interfaces aren't the best in the world... The maintainer/driver/hardware combination is a bit quirky at times but it does not work too bad. :o) -- Ueimor Anybody got a battery for my Ultra 10 ? - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: r8169 ethernet bonding problems
Francois Romieu [EMAIL PROTECTED] : [...] A tcpdump trace 1) before promiscuous mode 2) after promiscuous mode 3) after link failure (i.e. failover does not work) would be welcome. A 'lspci -vx' will be needed too. There are different kinds of 8169. -- Ueimor Anybody got a battery for my Ultra 10 ? - To unsubscribe from this list: send the line unsubscribe netdev in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
