Re: DPAA Ethernet problems with mainstream Linux kernels
Hi Jamie, Many thanks for your effort. If you need a new kernel for testing please let me know. Cheers, Christian Sent from my iPhone > On 18. Jan 2018, at 02:59, Jamie Krueger > wrote: > > Hi Madalin, > > On 01/16/2018 11:33 AM, Madalin-cristian Bucur wrote: > > Here are some results from testing on the X5000/20 with 4.15.0-rc8 > (w/CONFIG_FSL_PAMU disabled): > > In my tests I can now get the interface to obtain an IP Address via DHCP, > at least *part* of the time that is. > > Here is a link to a screenshot of a Wireshark capture which shows > a successful DHCP request/response for the X5000/20, followed by some > failed ping attempts. > > (This traffic was captured on an external machine on the same subnet, and not > from the X5000/20 itself) > http://bitbybitsoftwaregroup.com/share/X5000-DHCP-Answered-Wireshark.png > > Additionally, here is a link to the export of the packet data shown in the > above image: > http://bitbybitsoftwaregroup.com/share/Wireshark-X5000-DHCP-Success.html > > As you can see by this example, even when the interface does obtain an IP > address via > DHCP (about half the time), subsequent traffic still fails to receive > responses. > > In this example, I attempt to ping the gateway (192.168.1.1), and one or two > other > local addresses, and nothing pings back (even when trying Skateman's > suggestion > of expanding the buffers for the interface - > > "A workaround to keep the nic alive a bit longer is the following command > ip link set eth0 qlen 1" > > For reference: > > I have attached the Kernel config that the version I tested was built with. > Also, I have attached the dmesg output I am currently seeing. > > -- > Best Regards, > > Jamie Krueger > BITbyBIT Software Group LLC > > >
Re: DPAA Ethernet problems with mainstream Linux kernels
Hi All, I compiled the RC8 of kernel 4.15 for the X5000 without PAMU support today. Download: http://www.xenosoft.de/uImage_without_pamu.tar.gz Please test it on your AmigaOne X5000. Thanks, Christian On 16 January 2018 at 6:33PM, Madalin-cristian Bucur wrote: The PAMU related errors may be relevant to the issue, if you have incorrect settings you may have no traffic passing through. The PAMU configuration should be made by the bootloader. Can you try to disable CONFIG_FSL_PAMU? Madalin
RE: DPAA Ethernet problems with mainstream Linux kernels
> -Original Message- > From: Darren Stevens [mailto:dar...@stevens-zone.net] > Sent: Tuesday, January 16, 2018 12:40 AM > To: Madalin-cristian Bucur > Cc: Jamie Krueger ; linuxppc- > d...@lists.ozlabs.org; netdev@vger.kernel.org > Subject: Re: DPAA Ethernet problems with mainstream Linux kernels > > Hello Madalin-cristian > > On 15/01/2018, Madalin-cristian Bucur wrote: > >> > The device tree that you mention, cyrus_p5020.eth.dts is not found in > >> > the Linux kernel sources. The cyrus_p5020.dts file from the fsl ppc > >> > device tree folder does not include the PHY information for the DPAA > >> > interfaces. The problems that you experience may be caused by some > >> > issues with the PHY configuration (i.e. internal delay). > >> The cyrus_p5020.eth.dts is a modified version of the cyrus_p5020.dts, > >> which of course was based off the original p5020ds.dts file. As you > >> noted, the current cyrus_p5020.dts file is incomplete, and does not > >> map the Ethernet connections properly. > > This is because the current linux kernel version of cyrus_p5020.dts > includes 'p5020si-pre.dtsi' and 'p5020si-post.dtsi' include files, which > orginally gave us working ethernet (when we used the SDK kernel) However > at some point you moved the ethernet aliases from the board dts file to > the p5020si-pre.dtsi file breaking the linkages for our board. > > cyrus-pre.dtsi is simply p5020si-pre.dtsi with only the correct aliases > in. > > >> ** I have attached both the cyrus_p5020.eth.dts and cyrus-pre.dtsi > >> files with this email for comparison. Please let me know if you > see > >> any corrections that should be made to either file. > > > > At a first glance they look fine to me. > > That's good to know. > > >> I have started testing along that line, using Wireshark to view the > >> traffic on the X5000/20 itself, and from another machine connected > >> on the same subnet. So far (as indicated by some details of in my > >> initial email), I can see outgoing broadcast requests (for DHCP) > >> being sent out from the X5000/20, and these requests are correctly > >> constructed and visible outside the X5000/20. > >> > >> However, no responses to the DHCP broadcasts appear to reach > >> to X5000/20's DPAA Ethernet. I will need to setup some further > >> tests to determine if the DHCP server saw the requests and responded > >> to them. (I assume the DHCP server is getting them, and responding, > >> as I can always get a successful DHCP response to the X5000/20 > >> when using an add-on Ethernet PICe card on the same subnet). > > This matches what I see, the interface I have connected to the network > shows an increasing number of transmitted packets, but no received ones. > > Jamie also noticed the following error in dmesg (right after the ethernet > port becomes active) > > [4.112165] fsl_dpa dpaa-ethernet.0 eth0: Probed interface eth0 > [4.116616] fsl_dpa dpaa-ethernet.1 eth1: Probed interface eth1 > [ ... ] > [ 106.501055] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready > [ 106.570944] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready > [ 106.605044] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready > [ 106.674918] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready > [ 108.702771] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > [ 109.032798] fsl-pamu: pamu_av_isr: access violation interrupt > [ 109.032806] fsl-pamu: pamu_av_isr: POES1= > [ 109.032808] fsl-pamu: pamu_av_isr: POES2= > [ 109.032809] fsl-pamu: pamu_av_isr: AVS1=002d0081 > [ 109.032811] fsl-pamu: pamu_av_isr: AVS2=0081 > [ 109.032813] fsl-pamu: pamu_av_isr: AVA=0001f1328000 > [ 109.032815] fsl-pamu: pamu_av_isr: UDAD=002d0001 > [ 109.032817] fsl-pamu: pamu_av_isr: POEA= > > I haven't seen this anywhere else, and wondered if it is relevant. > > Regards > Darren The PAMU related errors may be relevant to the issue, if you have incorrect settings you may have no traffic passing through. The PAMU configuration should be made by the bootloader. Can you try to disable CONFIG_FSL_PAMU? Madalin
Re: DPAA Ethernet problems with mainstream Linux kernels
Hello Madalin-cristian On 15/01/2018, Madalin-cristian Bucur wrote: >> > The device tree that you mention, cyrus_p5020.eth.dts is not found in >> > the Linux kernel sources. The cyrus_p5020.dts file from the fsl ppc >> > device tree folder does not include the PHY information for the DPAA >> > interfaces. The problems that you experience may be caused by some >> > issues with the PHY configuration (i.e. internal delay). >> The cyrus_p5020.eth.dts is a modified version of the cyrus_p5020.dts, >> which of course was based off the original p5020ds.dts file. As you >> noted, the current cyrus_p5020.dts file is incomplete, and does not >> map the Ethernet connections properly. This is because the current linux kernel version of cyrus_p5020.dts includes 'p5020si-pre.dtsi' and 'p5020si-post.dtsi' include files, which orginally gave us working ethernet (when we used the SDK kernel) However at some point you moved the ethernet aliases from the board dts file to the p5020si-pre.dtsi file breaking the linkages for our board. cyrus-pre.dtsi is simply p5020si-pre.dtsi with only the correct aliases in. >> ** I have attached both the cyrus_p5020.eth.dts and cyrus-pre.dtsi >> files with this email for comparison. Please let me know if you see >> any corrections that should be made to either file. > > At a first glance they look fine to me. That's good to know. >> I have started testing along that line, using Wireshark to view the >> traffic on the X5000/20 itself, and from another machine connected >> on the same subnet. So far (as indicated by some details of in my >> initial email), I can see outgoing broadcast requests (for DHCP) >> being sent out from the X5000/20, and these requests are correctly >> constructed and visible outside the X5000/20. >> >> However, no responses to the DHCP broadcasts appear to reach >> to X5000/20's DPAA Ethernet. I will need to setup some further >> tests to determine if the DHCP server saw the requests and responded >> to them. (I assume the DHCP server is getting them, and responding, >> as I can always get a successful DHCP response to the X5000/20 >> when using an add-on Ethernet PICe card on the same subnet). This matches what I see, the interface I have connected to the network shows an increasing number of transmitted packets, but no received ones. Jamie also noticed the following error in dmesg (right after the ethernet port becomes active) [4.112165] fsl_dpa dpaa-ethernet.0 eth0: Probed interface eth0 [4.116616] fsl_dpa dpaa-ethernet.1 eth1: Probed interface eth1 [ ... ] [ 106.501055] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready [ 106.570944] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready [ 106.605044] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 106.674918] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 108.702771] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 109.032798] fsl-pamu: pamu_av_isr: access violation interrupt [ 109.032806] fsl-pamu: pamu_av_isr: POES1= [ 109.032808] fsl-pamu: pamu_av_isr: POES2= [ 109.032809] fsl-pamu: pamu_av_isr: AVS1=002d0081 [ 109.032811] fsl-pamu: pamu_av_isr: AVS2=0081 [ 109.032813] fsl-pamu: pamu_av_isr: AVA=0001f1328000 [ 109.032815] fsl-pamu: pamu_av_isr: UDAD=002d0001 [ 109.032817] fsl-pamu: pamu_av_isr: POEA= I haven't seen this anywhere else, and wondered if it is relevant. Regards Darren
RE: DPAA Ethernet problems with mainstream Linux kernels
> -Original Message- > From: Jamie Krueger [mailto:ja...@bitbybitsoftwaregroup.com] > Sent: Friday, January 12, 2018 6:36 PM > To: Madalin-cristian Bucur ; linuxppc- > d...@lists.ozlabs.org > Cc: netdev@vger.kernel.org > Subject: Re: DPAA Ethernet problems with mainstream Linux kernels > > On 01/12/2018 08:22 AM, Madalin-cristian Bucur wrote: > >> -Original Message- > >> From: Linuxppc-dev [mailto:linuxppc-dev- > >> bounces+madalin.bucur=nxp@lists.ozlabs.org] On Behalf Of Jamie > Krueger > >> Sent: Wednesday, January 10, 2018 5:57 PM > >> To: linuxppc-...@lists.ozlabs.org > >> Subject: DPAA Ethernet problems with mainstream Linux kernels > >> > >> Hello all @ linuxppc-dev, > >> > >> I have been working with a team of people maintaining PowerPC > >> Linux for the new AmigaONE X5000/20 (a Freescale p5020 SoC based > >> machine). > >> > >> We are trying to determine why the submitted Data Path Acceleration > >> Architecture (DPAA) Ethernet Driver is not fully functional with > >> the mainstream Linux kernels. > > Hi Jamie, > Hi Madalin, > > We are testing the DPAA driver on several DS and RDB platforms and it > > is working properly. The issues you encounter with it on the X5000/20 > > are likely caused by some issues specific to that particular platform. > It is good to hear that the DPAA driver is functioning correctly > on the reference platforms. I am positive you are correct that > the issue is the difference in implementation on the X5000/20 > (Cyrus) motherboard, as compared to the reference boards. > > Can you verify which Linux Kernel sources your tests are being > performed on? We have been testing using the mainstream > Linux sources up to linux-4.15-rc6 thus far. Latest run is on 4.15.0-rc7-00200-gc92a9a4. > > The device tree that you mention, cyrus_p5020.eth.dts is not found in > > the Linux kernel sources. The cyrus_p5020.dts file from the fsl ppc > > device tree folder does not include the PHY information for the DPAA > > interfaces. The problems that you experience may be caused by some > > issues with the PHY configuration (i.e. internal delay). > The cyrus_p5020.eth.dts is a modified version of the cyrus_p5020.dts, > which of course was based off the original p5020ds.dts file. As you > noted, the current cyrus_p5020.dts file is incomplete, and does not > map the Ethernet connections properly. > > The cyrus_p5020.eth.dts file, along with it's cyrus-pre.dtsi dependent > file, are an attempt to correctly define the Ethernet hardware, as it is > implemented on the X5000/20. > > ** I have attached both the cyrus_p5020.eth.dts and cyrus-pre.dtsi > files with this email for comparison. Please let me know if you see > any corrections that should be made to either file. At a first glance they look fine to me. > I am not sure what PHY hardware/configuration you are using on the > DS and RDB platforms, but I can confirm that AmigaONE X5000/20 > (Cyrus Motherboard with p5020 SoC), has dTSEC 4 and dTSEC 5 > wired to two Micrel KSZ9021RN Gigabit Ethernet PHYs, using the > RGMII protocol. Since it's RGMII, I think you should look into RGMII internal delay requirements for this board. > > I suggest > > that you connect the DPAA interface to a traffic analyzer or directly > > to another device on which you can capture the incoming traffic and > > check that the received frames are correct. > I have started testing along that line, using Wireshark to view the > traffic on the X5000/20 itself, and from another machine connected > on the same subnet. So far (as indicated by some details of in my > initial email), I can see outgoing broadcast requests (for DHCP) > being sent out from the X5000/20, and these requests are correctly > constructed and visible outside the X5000/20. > > However, no responses to the DHCP broadcasts appear to reach > to X5000/20's DPAA Ethernet. I will need to setup some further > tests to determine if the DHCP server saw the requests and responded > to them. (I assume the DHCP server is getting them, and responding, > as I can always get a successful DHCP response to the X5000/20 > when using an add-on Ethernet PICe card on the same subnet). > > I will setup some more direct machine-to-machine testing to > see what else I can glean from the network traffic. That will provide more clarity to the actual issue. > Please have a look at the attached dts files, maybe there is something > obvious there we are not seeing. > > Also, given that the X5000/20 uses Micrel KSZ9021RN PHYs in RGMII > mode, what changes to the DPAA hardware con
Re: DPAA Ethernet problems with mainstream Linux kernels
On 01/12/2018 08:22 AM, Madalin-cristian Bucur wrote: -Original Message- From: Linuxppc-dev [mailto:linuxppc-dev- bounces+madalin.bucur=nxp@lists.ozlabs.org] On Behalf Of Jamie Krueger Sent: Wednesday, January 10, 2018 5:57 PM To: linuxppc-...@lists.ozlabs.org Subject: DPAA Ethernet problems with mainstream Linux kernels Hello all @ linuxppc-dev, I have been working with a team of people maintaining PowerPC Linux for the new AmigaONE X5000/20 (a Freescale p5020 SoC based machine). We are trying to determine why the submitted Data Path Acceleration Architecture (DPAA) Ethernet Driver is not fully functional with the mainstream Linux kernels. Hi Jamie, Hi Madalin, We are testing the DPAA driver on several DS and RDB platforms and it is working properly. The issues you encounter with it on the X5000/20 are likely caused by some issues specific to that particular platform. It is good to hear that the DPAA driver is functioning correctly on the reference platforms. I am positive you are correct that the issue is the difference in implementation on the X5000/20 (Cyrus) motherboard, as compared to the reference boards. Can you verify which Linux Kernel sources your tests are being performed on? We have been testing using the mainstream Linux sources up to linux-4.15-rc6 thus far. The device tree that you mention, cyrus_p5020.eth.dts is not found in the Linux kernel sources. The cyrus_p5020.dts file from the fsl ppc device tree folder does not include the PHY information for the DPAA interfaces. The problems that you experience may be caused by some issues with the PHY configuration (i.e. internal delay). The cyrus_p5020.eth.dts is a modified version of the cyrus_p5020.dts, which of course was based off the original p5020ds.dts file. As you noted, the current cyrus_p5020.dts file is incomplete, and does not map the Ethernet connections properly. The cyrus_p5020.eth.dts file, along with it's cyrus-pre.dtsi dependent file, are an attempt to correctly define the Ethernet hardware, as it is implemented on the X5000/20. ** I have attached both the cyrus_p5020.eth.dts and cyrus-pre.dtsi files with this email for comparison. Please let me know if you see any corrections that should be made to either file. I am not sure what PHY hardware/configuration you are using on the DS and RDB platforms, but I can confirm that AmigaONE X5000/20 (Cyrus Motherboard with p5020 SoC), has dTSEC 4 and dTSEC 5 wired to two Micrel KSZ9021RN Gigabit Ethernet PHYs, using the RGMII protocol. I suggest that you connect the DPAA interface to a traffic analyzer or directly to another device on which you can capture the incoming traffic and check that the received frames are correct. I have started testing along that line, using Wireshark to view the traffic on the X5000/20 itself, and from another machine connected on the same subnet. So far (as indicated by some details of in my initial email), I can see outgoing broadcast requests (for DHCP) being sent out from the X5000/20, and these requests are correctly constructed and visible outside the X5000/20. However, no responses to the DHCP broadcasts appear to reach to X5000/20's DPAA Ethernet. I will need to setup some further tests to determine if the DHCP server saw the requests and responded to them. (I assume the DHCP server is getting them, and responding, as I can always get a successful DHCP response to the X5000/20 when using an add-on Ethernet PICe card on the same subnet). I will setup some more direct machine-to-machine testing to see what else I can glean from the network traffic. Please have a look at the attached dts files, maybe there is something obvious there we are not seeing. Also, given that the X5000/20 uses Micrel KSZ9021RN PHYs in RGMII mode, what changes to the DPAA hardware configuration should we expect to see so that the DPAA is configured to talk to them? Madalin -- Best Regards, Jamie Krueger BITbyBIT Software Group LLC Here is the results from my latest tests. They were performed using the linux-4.10.17 ppc64, since that represents when the DPAA Ethernet code was introduced. Similar tests, with similar results, were also performed using the latest Linux kernels: linux-4.15-rc5 linux-4.15-rc6 linux-4.15-rc7 (Hence the reason for falling back to test the kernel right after the introduction of the DPAA Ethernet driver sources) --- All Kernel builds had the DPAA Ethernet enabled in the kernel, and are using the correct cyrus_p5020.eth.dtb device tree file (for use on the X5000/20). The results are quite similar for all kernels in regards to the DPAA Ethernet. All tested kernels setup the two Ethernet interfaces correctly as eth0 and eth1, and pull the correct MAC addresses from U-Boot environment variables ethaddr and eth1addr respectively. So at this point Linux has what it believes is fully configured hardware, waiting to have an IP Address/Netmask/Gateway to be set and to bring the interface online. How
RE: DPAA Ethernet problems with mainstream Linux kernels
> -Original Message- > From: Linuxppc-dev [mailto:linuxppc-dev- > bounces+madalin.bucur=nxp@lists.ozlabs.org] On Behalf Of Jamie Krueger > Sent: Wednesday, January 10, 2018 5:57 PM > To: linuxppc-...@lists.ozlabs.org > Subject: DPAA Ethernet problems with mainstream Linux kernels > > Hello all @ linuxppc-dev, > > I have been working with a team of people maintaining PowerPC > Linux for the new AmigaONE X5000/20 (a Freescale p5020 SoC based > machine). > > We are trying to determine why the submitted Data Path Acceleration > Architecture (DPAA) Ethernet Driver is not fully functional with > the mainstream Linux kernels. Hi Jamie, We are testing the DPAA driver on several DS and RDB platforms and it is working properly. The issues you encounter with it on the X5000/20 are likely caused by some issues specific to that particular platform. The device tree that you mention, cyrus_p5020.eth.dts is not found in the Linux kernel sources. The cyrus_p5020.dts file from the fsl ppc device tree folder does not include the PHY information for the DPAA interfaces. The problems that you experience may be caused by some issues with the PHY configuration (i.e. internal delay). I suggest that you connect the DPAA interface to a traffic analyzer or directly to another device on which you can capture the incoming traffic and check that the received frames are correct. Madalin > Here is the results from my latest tests. They were performed using > the linux-4.10.17 ppc64, since that represents when the DPAA Ethernet > code was introduced. > > Similar tests, with similar results, were also performed > using the latest Linux kernels: > > linux-4.15-rc5 > linux-4.15-rc6 > linux-4.15-rc7 > > (Hence the reason for falling back to test the kernel right > after the introduction of the DPAA Ethernet driver sources) > > --- > > All Kernel builds had the DPAA Ethernet enabled in the kernel, > and are using the correct cyrus_p5020.eth.dtb device tree file > (for use on the X5000/20). > > The results are quite similar for all kernels in regards to the DPAA > Ethernet. > > All tested kernels setup the two Ethernet interfaces correctly > as eth0 and eth1, and pull the correct MAC addresses from U-Boot > environment variables ethaddr and eth1addr respectively. > > So at this point Linux has what it believes is fully configured > hardware, waiting to have an IP Address/Netmask/Gateway > to be set and to bring the interface online. > > However, all attempts to communicate with the outside world > do not make it out the physical (PHY) hardware - or do they? > > ** The following results were captured under linux-4.10.17 ** > > When I bring the interface up using a static address, in this case > 192.168.1.21, I see the following (NOTE TX bytes says 154.0 KB, > while RX bytes says 0.0 B): > > jamie@X5000-Linux:$ ifconfig > eth0 Link encap:Ethernet HWaddr 00:80:10:11:11:11 > inet addr:192.168.1.21 Bcast:192.168.1.255 Mask:255.255.255.0 > inet6 addr: fe80::280:10ff:fe11:/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:1428 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:154066 (154.0 KB) > Memory:fe4e6000-fe4e6fff > > eth1 Link encap:Ethernet HWaddr 00:80:10:22:22:22 > UP BROADCAST MULTICAST MTU:1500 Metric:1 > RX packets:0 errors:0 dropped:0 overruns:0 frame:0 > TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) > Memory:fe4e8000-fe4e8fff > > lo Link encap:Local Loopback > inet addr:127.0.0.1 Mask:255.0.0.0 > inet6 addr: ::1/128 Scope:Host > UP LOOPBACK RUNNING MTU:65536 Metric:1 > RX packets:1869 errors:0 dropped:0 overruns:0 frame:0 > TX packets:1869 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:156932 (156.9 KB) TX bytes:156932 (156.9 KB) > > Checking the routing table, everything looks fine there: > > jamie@X5000-Linux:$ netstat -r > Kernel IP routing table > Destination Gateway Genmask Flags MSS Window irtt > Iface > default 192.168.1.1 0.0.0.0 UG 0 0 0 > eth0 > link-local * 255.255.0.0 U 0 0 0 > eth0 > 192.168.1.0 * 255.255.255.0 U 0 0 0 > eth0 > > Attempting to PING the interface itself works: > > jamie@X5000-Linux:$ ping 192.168.1.21 > PING 192.168.1.21 (192.168.1.21) 56(84) bytes of data. > 64 bytes from 192.168.1.21: icmp_seq=1 ttl=64 time=0.037 ms > 64 bytes from 192.168.1.21: icmp_seq=2 ttl=64 time=0.045 ms > 64 bytes from 192.168.1.21: icmp_seq=3 ttl=64 time=0.033
