Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Quoting Jonas Smedegaard (2022-06-09 12:25:01) > This bug was fixed since u-boot 2022.04 - specifically in git commit > f11513d9: https://source.denx.de/u-boot/u-boot/-/commit/f11513d Correction - the actual fix was the later git commit 85da5587: https://source.denx.de/u-boot/u-boot/-/commit/85da558 - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Quoting Jonas Smedegaard (2019-04-26 09:54:37) > Quoting Sunil Mohan Adapa (2019-04-25 21:30:03) > > > Were your Lime2 boards connected with a cross-over cable or via a > > > switch during those tests? > > > > Lime2 was connected to a laptop via a cross-over cable (actually > > regular cable but the hardware actually detects cross-over setup and > > automatically swaps TX/RX). > > Thanks. Makes good sense to me now (sorry for being dense). It seems wiring while testing _is_ relevant after all: Concretely, wiring affects MDI/MDI-X, commonly emulated transparently in both hosts and switches nowadays (because it is an optional part of the standard for gigabit ethernet), with no need for actually using a "cross-over cable". I do not suspect any flaws in MDI/MDI-X handling itself, but indirectly the _need_ for NDI-X matters anyway: The cause for the packet loss issues is likely a timing issue. Ethernet is tied to a clock signal fed from one end of the wiring - the "master". When using a switch the switch end of the wiring becomes master, but in "cross-over" wiring (no matter if a cross-over cable is used or whichever of the host PHYs emulate cross-over by flipping from the normal MDI to MDI-X) it is _undefined_ which end gets becomes master. It can seem more reliable to setup a minimal test involving only two hosts and a cable, but in reality that introduces less reliable results than using a switch in-between. Sunil: It would be helpful to know who from Olimex you talked to, so that we can try get back to them and figure out if their tests leading to 20% failure was done "head-to-head" (where it is undefined which end becomes master but perhaps just a fancier chipset at the peer end wins a random race 80% of the time), or they used a switch (where I cannot think of such obvious explanation for the 20% failure, and fall back on "20% of the chips are behaving differently than the rest"). What I am hoping for is to that all chips behave the same - that the 20% can be explained by the wiring in the test setup. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Here's what I know (the "should work" entries need confirmation): Lime2 rev.C --- * Uses Realtek rev.C PHY * Works fine in 1Gbit mode with Debian stable U-boot * Works fine in 1Gbit mode with Debian stable kernel * Possibly still sold as some no-eMMC no-flash options Lime2 rev.G --- * Uses Realtek rev.E PHY * Sold as FreedomBox Edition * Possibly still sold as some no-flash options * Works in 100Mbit mode with Debian-stable kernel and u-boot * Works in 1Gbit mode with Debian-stable kernel and custom U-boot patched to set TX_DELAY=4 Lime rev.H -- * Uses Micrel PHY * Sold (at least) as flash options * Works in 10Mbit mode with Debian stable kernel and u-boot * Should work in 1Gbit mode with Debian-backports kernel v5.2 * Should work in 1Gbit mode with custom kernel patched to to set CONFIG_MICREL_PHY=y (already done with Debian kernels) and to include https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3aed3e2 (applied upstream since v5.2) With systemd-networkd, all lime2 boards work with this added as file /etc/systemd/network/90-ethernet.link (and rev.G boards work also with 100baset entries uncommented): [Match] Driver=st_gmac [Link] NamePolicy=keep kernel database onboard slot path MACAddressPolicy=persistent Advertise=10baset-half Advertise=10baset-full #Advertise=100baset-half #Advertise=100baset-full #Advertise=1000baset-half #Advertise=1000baset-full - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
On 27/04/19 2:46 am, Andreas B. Mundt wrote: > Hi all, > > On Fri, Apr 26, 2019 at 09:54:37AM +0200, Jonas Smedegaard wrote: >> >> So a summary of the tests is this: >> >> Board, cable mode, patchset \ Measured speed in Mbit/sec >> rev.G2 1Gbit mode pristine:14 & 722 >> rev.G2 1Gbit mode TX_DELAY=4: 428 & 774 >> rev.C 1Gbit pristine: 1 & 526 >> rev.C 100Mbit pristine:94 & 94 >> rev.C 1Gbit TX_DELAY=4:61 & 141 >> rev.C 100Mbit TX_DELAY=4: 89 & 89 >> > > I tried to reproduce the results for my rev.G2 board (2017), but could > not reproduce the bad performance. My results are: Oops! Forget to mention an important detail. Only about 20% of the Rev.G2 boards that Olimex tested (out of a large number) were facing this issue. We were also not able to reproduce the problem on one of the Rev.G2 boards. However, Olimex assures us that the problem is real. They mentioned that tolerances ranges of various hardware components could be the reason for only some boards facing this issue and not all. Setting TX_DELAY=4 works on 100% of all Rev. G2 boards. When releasing FreedomBox Pioneer Edition this week, we had to unfortunately ship with a patched version of u-boot with TX_DELAY=4. -- Sunil signature.asc Description: OpenPGP digital signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Hi all, On Fri, Apr 26, 2019 at 09:54:37AM +0200, Jonas Smedegaard wrote: > > So a summary of the tests is this: > > Board, cable mode, patchset \ Measured speed in Mbit/sec > rev.G2 1Gbit mode pristine:14 & 722 > rev.G2 1Gbit mode TX_DELAY=4: 428 & 774 > rev.C 1Gbit pristine: 1 & 526 > rev.C 100Mbit pristine:94 & 94 > rev.C 1Gbit TX_DELAY=4:61 & 141 > rev.C 100Mbit TX_DELAY=4: 89 & 89 > I tried to reproduce the results for my rev.G2 board (2017), but could not reproduce the bad performance. My results are: ansible@blackbox:~$ iperf3 -c 192.168.122.1 Connecting to host 192.168.122.1, port 5201 [ 5] local 192.168.122.223 port 60618 connected to 192.168.122.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 70.0 MBytes 587 Mbits/sec0752 KBytes [ 5] 1.00-2.00 sec 68.9 MBytes 578 Mbits/sec0923 KBytes [ 5] 2.00-3.00 sec 67.4 MBytes 566 Mbits/sec0 1012 KBytes [ 5] 3.00-4.00 sec 65.7 MBytes 550 Mbits/sec0 1.15 MBytes [ 5] 4.00-5.00 sec 65.0 MBytes 546 Mbits/sec0 1.21 MBytes [ 5] 5.00-6.00 sec 68.8 MBytes 577 Mbits/sec 42904 KBytes [ 5] 6.00-7.00 sec 67.5 MBytes 564 Mbits/sec0929 KBytes [ 5] 7.00-8.01 sec 52.5 MBytes 437 Mbits/sec0929 KBytes [ 5] 8.01-9.00 sec 55.0 MBytes 465 Mbits/sec0 1.08 MBytes [ 5] 9.00-10.01 sec 56.2 MBytes 468 Mbits/sec 18803 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 637 MBytes 534 Mbits/sec 60 sender [ 5] 0.00-10.05 sec 637 MBytes 532 Mbits/sec receiver iperf Done. ansible@blackbox:~$ iperf3 -c 192.168.122.1 -R Connecting to host 192.168.122.1, port 5201 Reverse mode, remote host 192.168.122.1 is sending [ 5] local 192.168.122.223 port 60622 connected to 192.168.122.1 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 94.0 MBytes 788 Mbits/sec [ 5] 1.00-2.00 sec 106 MBytes 888 Mbits/sec [ 5] 2.00-3.00 sec 108 MBytes 906 Mbits/sec [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec [ 5] 6.00-7.00 sec 111 MBytes 932 Mbits/sec [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec [ 5] 8.00-9.00 sec 108 MBytes 908 Mbits/sec [ 5] 9.00-10.00 sec 109 MBytes 917 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.04 sec 1.06 GBytes 906 Mbits/sec 179 sender [ 5] 0.00-10.00 sec 1.06 GBytes 907 Mbits/sec receiver iperf Done. U-boot is the version from experimental, 2019.04+dfsg-2. Regards, Andi
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Quoting Jonas Smedegaard (2019-04-26 09:54:37) > Would be good to have tests explicitly in master vs. slave mode. Seems master/slave can at least be probed (and possibly forced) in linux with ethtool, and in u-boot with mii - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Quoting Sunil Mohan Adapa (2019-04-25 21:30:03) > > > > It seems you forgot to attach the mentioned Olimex report. > > The report is in the inline section "Report from Olimex team (with > Rev.G2)". I used term 'attached' loosely. Ahh: You _quoted_ Olimex from "Report from Olimex team (with Rev.G2)" down to and including the iperf tests on ip 192.168.0.60, and then you _appended_ tests of your own for a rev.C board on ip 10.42.1.1. > > Do I understand you correctly that the attached patch is what Olimex > > propose but that you do *not* recommended to use it as-is because it > > badly affects older boards? > > Olimex has kindly provided us the patch so that we can create a fully > working u-boot build for Lime2 Rev.G2 board. They did not imply that > the patch was suitable for other boards as well. Understood (now). Thanks. > > Were your Lime2 boards connected with a cross-over cable or via a > > switch during those tests? > > Lime2 was connected to a laptop via a cross-over cable (actually > regular cable but the hardware actually detects cross-over setup and > automatically swaps TX/RX). Thanks. Makes good sense to me now (sorry for being dense). So a summary of the tests is this: Board, cable mode, patchset \ Measured speed in Mbit/sec rev.G2 1Gbit mode pristine:14 & 722 rev.G2 1Gbit mode TX_DELAY=4: 428 & 774 rev.C 1Gbit pristine: 1 & 526 rev.C 100Mbit pristine:94 & 94 rev.C 1Gbit TX_DELAY=4:61 & 141 rev.C 100Mbit TX_DELAY=4: 89 & 89 Would be good to have tests explicitly in master vs. slave mode. Would be good to have tests with TX_DELAY=2 and TX_DELAY=3 (the former seems to be what Olimex set themselves for rev.G, and the latter is what some u-boot hacker has noted as working on linux-sunxi wiki). Would be good to have tests for rev.H and/or rev. K board (to understand if changes cause regressions on that board - yes I am aware that some FreedomBox reports indicate that board being broken too but we desperately need detailed knowledge on that filed in the Debian BTS!). - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
> > It seems you forgot to attach the mentioned Olimex report. The report is in the inline section "Report from Olimex team (with Rev.G2)". I used term 'attached' loosely. > > Do I understand you correctly that the attached patch is what Olimex > propose but that you do *not* recommended to use it as-is because it > badly affects older boards? Olimex has kindly provided us the patch so that we can create a fully working u-boot build for Lime2 Rev.G2 board. They did not imply that the patch was suitable for other boards as well. > > Were your Lime2 boards connected with a cross-over cable or via a switch > during those tests? Lime2 was connected to a laptop via a cross-over cable (actually regular cable but the hardware actually detects cross-over setup and automatically swaps TX/RX). [...] -- Sunil signature.asc Description: OpenPGP digital signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
The following is the patch Olimex has applied on u-boot for the images that they build. It is meant to work for all hardware revisions of Lime2. https://github.com/OLIMEX/OLINUXINO/blob/master/SOFTWARE/A20/A20-build-3.4.103-release-7/a20-phy_1000_100-dram.patch It does not seem suitable for non-Lime2 boards and may need changes to before it can be submitted upstream (or Debian). Thanks, -- Sunil signature.asc Description: OpenPGP digital signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
It seems mainline u-boot bootstrao all lime2 devices equally, whereas Olimex fork of u-boot treats "newer than G and "newer than E" specially: https://github.com/OLIMEX/u-boot/commit/6c32a3c9d31432884751966fcb0f15b1fd930446 * Realtek rev.C PHY (board rev.C) get no tweak (but see bug#845128) * Realtek rev.E PHY (board rev.G,G1,G2) get TX_DELAY=2 * Micrel PHY (board rev.H,K) get TX_DELAY=4 ...as pointed out on irc up until here: https://freenode.irclog.whitequark.org/linux-sunxi/2019-04-25#24483567 - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Hi Sunil, > Olimex team working with FreedomBox team have noticed a significant > slowdown in Gigabit Ethernet when transmitting. Original report from > Olimex team is attached. This is applicable for hardware Rev.G2 of the > A20 OLinuXino Lime2 board. It seems you forgot to attach the mentioned Olimex report. Do I understand you correctly that the attached patch is what Olimex propose but that you do *not* recommended to use it as-is because it badly affects older boards? Were your Lime2 boards connected with a cross-over cable or via a switch during those tests? I am no expert on these things, but did some digging...: Older Lime2 use RTL8211CL-GR and Rev.G onwards use RTL8211E-VB-CG1, according to https://github.com/OLIMEX/OLINUXINO/tree/master/HARDWARE/A20-OLinuXino-LIME2 Lime2 RTL8211C has a buggy PLL, and u-boot since 2016.11+dfsg1-2 force RTL8211C (but *not* RTL8211E) to run as master when in gigabit mode, according to u-boot commit cebf3f5 Upstream discussion and tests when introducing CONFIG_GMAC_TX_DELAY: https://lists.denx.de/pipermail/u-boot/2016-March/248498.html Above upstream tests most likely used only the older RTL8211C PHY, as rev.G was seemingly introduced mid 2016 at earliest, according to https://olimex.wordpress.com/2016/12/08/a20-olinuxino-lime2-now-with-pcb-revision-g/ and https://olimex.wordpress.com/2016/05/04/lime2-get-better-now-with-emmc-flash-a20-olinuxino-lime2-emmc/ and (vaguely) https://github.com/OLIMEX/OLINUXINO/blob/master/HARDWARE/A20-OLinuXino-LIME2/hardware_revision_changes_log.txt#L61 - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private signature.asc Description: signature
Bug#927397: u-boot: Very poor ethernet performance on A20 OLinuXino Lime2 Rev.G2
Source: u-boot Version: 2019.01+dfsg-4 Severity: important -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Dear Maintainer, Olimex team working with FreedomBox team have noticed a significant slowdown in Gigabit Ethernet when transmitting. Original report from Olimex team is attached. This is applicable for hardware Rev.G2 of the A20 OLinuXino Lime2 board. Olimex team have sent a patch that fixes the issue for Rev.G2 (attached). Multiple team members from FreedomBox have confirmed that the patch fixes transmit performance issue for Rev.G2 and yields good transmit/receive performance. However this patch makes the situation worse for Rev.C (my test results attached). I believe this issue is separate from #845128 (will post more information there) which already causes *receive* performance to be bad on Rev.C. = Report from Olimex team (with Rev.G2) = Our test shows that speed in TX side is very slow and unstable. The problem is in the wrong clock delay settings. We corrected this settings in our image and recommend you to change the TX delay too. You have to change value of CCM_GMAC_CTRL_TX_CLK_DELAY register on address 0x01c20164 from 0x0006 to 0x1006. This should be changed in the u-boot sources and the image should be compiled again. - -- Tests with original image. Unchaged CCM_GMAC_CTRL_TX_CLK_DELAY value(0006) - -- => md.l 0x01c20164 1 01c20164: 0006 admin@freedombox:~$ iperf3 -c 192.168.0.60 -p 2001 Connecting to host 192.168.0.60, port 2001 [ 5] local 192.168.0.135 port 57448 connected to 192.168.0.60 port 2001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 375 KBytes 3.07 Mbits/sec 32 4.24 KBytes [ 5] 1.00-2.00 sec 1.67 MBytes 14.0 Mbits/sec 147 9.90 KBytes [ 5] 2.00-3.00 sec 4.49 MBytes 37.6 Mbits/sec 261 2.83 KBytes [ 5] 3.00-4.00 sec 669 KBytes 5.48 Mbits/sec 61 5.66 KBytes [ 5] 4.00-5.00 sec 2.78 MBytes 23.3 Mbits/sec 160 1.41 KBytes [ 5] 5.00-6.00 sec 829 KBytes 6.79 Mbits/sec 88 1.41 KBytes [ 5] 6.00-7.00 sec 1.47 MBytes 12.4 Mbits/sec 94 2.83 KBytes [ 5] 7.00-8.00 sec 1.55 MBytes 13.0 Mbits/sec 98 1.41 KBytes [ 5] 8.00-9.00 sec 2.47 MBytes 20.8 Mbits/sec 169 1.41 KBytes [ 5] 9.00-10.00 sec 700 KBytes 5.73 Mbits/sec 44 2.83 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 16.9 MBytes 14.2 Mbits/sec 1154 sender [ 5] 0.00-10.00 sec 16.8 MBytes 14.1 Mbits/sec receiver iperf Done. admin@freedombox:~$ iperf3 -c 192.168.0.60 -p 2001 -R Connecting to host 192.168.0.60, port 2001 Reverse mode, remote host 192.168.0.60 is sending [ 5] local 192.168.0.135 port 57452 connected to 192.168.0.60 port 2001 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 97.1 MBytes 814 Mbits/sec [ 5] 1.00-2.00 sec 98.2 MBytes 824 Mbits/sec [ 5] 2.00-3.00 sec 98.5 MBytes 826 Mbits/sec [ 5] 3.00-4.00 sec 48.6 MBytes 408 Mbits/sec [ 5] 4.00-5.00 sec 98.2 MBytes 824 Mbits/sec [ 5] 5.00-6.00 sec 97.9 MBytes 821 Mbits/sec [ 5] 6.00-7.00 sec 98.2 MBytes 824 Mbits/sec [ 5] 7.00-8.00 sec 97.7 MBytes 820 Mbits/sec [ 5] 8.00-9.00 sec 98.4 MBytes 826 Mbits/sec [ 5] 9.00-10.00 sec 27.5 MBytes 231 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 861 MBytes 722 Mbits/sec 58 sender [ 5] 0.00-10.00 sec 860 MBytes 722 Mbits/sec receiver iperf Done. - - Test with CCM_GMAC_CTRL_TX_CLK_DELAY value set to 1006 - - => md.l 0x01c20164 1 01c20164: 1006 admin@freedombox:~$ iperf3 -c 192.168.0.60 -p 2001 Connecting to host 192.168.0.60, port 2001 [ 5] local 192.168.0.135 port 53002 connected to 192.168.0.60 port 2001 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 47.5 MBytes 398 Mbits/sec0153 KBytes [ 5] 1.00-2.01 sec 53.8 MBytes 448 Mbits/sec0206 KBytes [ 5] 2.01-3.00 sec 32.8 MBytes 277 Mbits/sec1235 KBytes [ 5] 3.00-4.00 sec 67.5 MBytes 567 Mbits/sec0235 KBytes [ 5] 4.00-5.00 sec 56.2 MBytes 471 Mbits/sec0235 KBytes [ 5] 5.00-6.02 sec 48.8 MBytes 400 Mbits/sec0235 KBytes [ 5] 6.02-7.01 sec 47.5 MBytes 402 Mbits/sec0235 KBytes [ 5] 7.01-8.02 sec 48.8 MBytes
