Re: [casper] Receive path automatic link training on ROACH2 10GbE
Hi John, There is a few parts to this: The first thing is that the phy programming infrastructure is built into the new tcpborphserver with the ?phyprog katcp command. This version of tcpborphserver is contained in the romfs image. This can be extracted by mounting the romfs image as a loop device and then copying it out of /sbin $ mount -o loop roach2-root-phyprog-release-2015-04-01.romfs /mnt $ cd /mnt/sbin You can then run this version of tcpborphserver. Secondly, you will need the FPGA gateware which can also be found in the romfs in /boffiles. It's called r2_switch_tst_sys2_2014_May_20_0847.bof.gz You should unzip it and place it in the directory which tcpborphserver points to for bof files (default is /boffiles). The phy firmware binary obtained from Vitesse should also be placed in this directory. At this point, you could manually load the firmware by connecting to the new tcpborphserver via telnet. You should load the FPGA gateware with the ?progdev command and thereafter use the ?phyprog [mezz] [phy] command. Run this for each PHY on each mezzanine card, i.e ?progdev r2_switch_tst_sys2_2014_May_20_0847.bof ?phyprog 0 0 ?phyprog 0 1 ?phyprog 1 0 ?phyprog 1 1 You can use the ?phywatch [mezz] [phy] command to verify that the firmware is running. Issuing two consecutive ?phywatch commands should indicate whether the watchdog counter is incrementing and therefore confirm that the firmware is running. Run ?progdev to unload the FPGA again. There is also a script in the romfs at /etc/rc.physetup which can be run on the ROACH2 to perform this phy firmware loading once tcpborphserver is running. I hope this information helps you to get it up and running! Regards, Robin. On Wed, Apr 15, 2015 at 4:00 PM, John Ford jf...@nrao.edu wrote: Hi Robin. We currently use an NFS mounted file system. Can you tell me what I need to upgrade to make this work? Thanks! John Hi All, A new romfs containing the functionality to program the Vitesse VCS8488 PHYs on the ROACH2 10GbE mezzanine cards is available on Github at the following link: https://github.com/ska-sa/roach2_nfs_uboot/tree/master/boot and is called: roach2-root-phyprog-release-2015-04-01.romfs It contains an updated version of tcpborphserver, which includes the PHY programming commands, as well as scripts to load the PHY firmware at startup. As mentioned by Steve, the firmware is under NDA and can be obtained directly from Vitesse. Once the new romfs has been loaded via uboot, the following should be done for the upgrade to take effect (this only needs to be done once): - delete /var/.keep file. [*WARNING: this will delete all your previous /etc configurations*] - copy the vsc848x_EDC_FW_1_14.bin firmware file obtained from Vitesse into the /boffiles directory - reboot the ROACH2 On each subsequent reboot, the firmware will be loaded into each PHY's volatile memory. This process will also briefly program/deprogram the FPGA at startup and should complete within a minute or two. Alternatively, the PHYs can also be programmed manually by connecting to tcpborphserver and running the phyprog command. Regards, Robin van Wyk Software Engineer SKA-SA On Wed, Apr 15, 2015 at 1:39 PM, Stephen Dennehy sdenn...@ska.ac.za wrote: Hi All The Vitesse VCS8488 PHYs used on the 10GbE mezzanine cards on ROACH2 run, by default, with no Electronic Dispersion Compensation (EDC). They do however have provision for this to be enabled, by loading an application onto the PHY itself. We have tested this and found a dramatic decrease in receive path bit error rate. That's the good news. The bad news is that the application is under NDA from Vitesse, and can be accessed only after each user organisation signs a separate NDA directly with Vitesse. We asked if we could use one NDA for the whole CASPER community, but it was politely declined. The process is quite simple: - Register as a user on www.vitesse.com - Fill in the attached NDA form and send it to susan...@vitesse.com - Once approved, they will grant you access to the magical file vsc848x_EDC_FW_1_14.bin - Copy that file into the same directory as the bof files. - The latest version of romfs will look for the file at startup. If present and not yet loaded, it will load the application into volatile memory on the PHYs. If the file is not present, it will simply skip the loading and continue as before. The improvement in performance is significant, so it would probably be worth your while to do this, even though the NDA process is a bit cumbersome. *Loading instructions will follow shortly..* Thank you -- Steve Dennehy CBF Systems Engineer SKA-SA
Re: [casper] Regarding dram: Moving from ROACH1 to ROACH2
Hi Tim, If you don't mind sharing your design, I'll put it up on the Casper wiki, where i think it would be useful for others trying to use the dram. Cheers, Jack On Thu, 16 Apr 2015 7:19 am Madden, Timothy J. tmad...@aps.anl.gov wrote: Folks After reverse engineering the dram on ROACH2 here is what one should know. 1. The dram on roach2 is different from roach1. On roach1, the user can read and write 144 size words to the dram with the buss set not to 288 bits. On roach2, the data buss is always 288 bits, regardless of the use wide buss setting. 2. Of one wants to stream dram data to a DAC on a roach 1, for MKID applications we do this: Read every other clock cycle by toggling cmd ack on each clock. 144 bit words will steam out every clock that can drive the DAC. On roach2, we read 288 bit words every other clock cycle by toggling cmd ack on every clock. Then we use a mux to convert the 288 bit words down to a series of 144 bit words on every clock. The dram should be set to 200MHz, not 240MHz or something else. The CORE generator has all its constraints set up for 200MHz. Note that this 200MHz has nothing to do with your fabric clock, as it comes from a different PLL on the FPGA. The yellow block does not interface to the dram itself, but simply FIFOs. When you write to the dram yellow block, you write to FIFOs that cross the clock domains. For streaming 144 bit data to DACs, you are reading 288 bit data from the dram at 1/2 your fabric clock rate. This gives lots of time for the FIFOs to read from the dram in little 8 word bursts and still deal with the dram refresh. I post this because we got confused by the documentation on the casper website. Also, we are moving from ROACH to ROACH2, and did not know the little differences in the dram. Also, it is natural to assume that the yellow block talks directly to the xilinx ddr3 controller and the ports correspond to the UI interface documented in the xilinx memory generator docs. This is not the case. The yellow block reads and writes fifos, and the ddr stuff is all hidden. The dram is a nice design by the way. I am just pointing out some details to save other folks some time. Tim Madden Argonne Lab
Re: [casper] Regarding dram: Moving from ROACH1 to ROACH2
I will get on github soon. Argonne and Nist are sharing stuff already. T From: Jack Hickish [jackhick...@gmail.com] Sent: Thursday, April 16, 2015 10:08 AM To: Madden, Timothy J.; casper@lists.berkeley.edu Subject: Re: [casper] Regarding dram: Moving from ROACH1 to ROACH2 Hi Tim, If you don't mind sharing your design, I'll put it up on the Casper wiki, where i think it would be useful for others trying to use the dram. Cheers, Jack On Thu, 16 Apr 2015 7:19 am Madden, Timothy J. tmad...@aps.anl.govmailto:tmad...@aps.anl.gov wrote: Folks After reverse engineering the dram on ROACH2 here is what one should know. 1. The dram on roach2 is different from roach1. On roach1, the user can read and write 144 size words to the dram with the buss set not to 288 bits. On roach2, the data buss is always 288 bits, regardless of the use wide buss setting. 2. Of one wants to stream dram data to a DAC on a roach 1, for MKID applications we do this: Read every other clock cycle by toggling cmd ack on each clock. 144 bit words will steam out every clock that can drive the DAC. On roach2, we read 288 bit words every other clock cycle by toggling cmd ack on every clock. Then we use a mux to convert the 288 bit words down to a series of 144 bit words on every clock. The dram should be set to 200MHz, not 240MHz or something else. The CORE generator has all its constraints set up for 200MHz. Note that this 200MHz has nothing to do with your fabric clock, as it comes from a different PLL on the FPGA. The yellow block does not interface to the dram itself, but simply FIFOs. When you write to the dram yellow block, you write to FIFOs that cross the clock domains. For streaming 144 bit data to DACs, you are reading 288 bit data from the dram at 1/2 your fabric clock rate. This gives lots of time for the FIFOs to read from the dram in little 8 word bursts and still deal with the dram refresh. I post this because we got confused by the documentation on the casper website. Also, we are moving from ROACH to ROACH2, and did not know the little differences in the dram. Also, it is natural to assume that the yellow block talks directly to the xilinx ddr3 controller and the ports correspond to the UI interface documented in the xilinx memory generator docs. This is not the case. The yellow block reads and writes fifos, and the ddr stuff is all hidden. The dram is a nice design by the way. I am just pointing out some details to save other folks some time. Tim Madden Argonne Lab
[casper] Regarding dram: Moving from ROACH1 to ROACH2
Folks After reverse engineering the dram on ROACH2 here is what one should know. 1. The dram on roach2 is different from roach1. On roach1, the user can read and write 144 size words to the dram with the buss set not to 288 bits. On roach2, the data buss is always 288 bits, regardless of the use wide buss setting. 2. Of one wants to stream dram data to a DAC on a roach 1, for MKID applications we do this: Read every other clock cycle by toggling cmd ack on each clock. 144 bit words will steam out every clock that can drive the DAC. On roach2, we read 288 bit words every other clock cycle by toggling cmd ack on every clock. Then we use a mux to convert the 288 bit words down to a series of 144 bit words on every clock. The dram should be set to 200MHz, not 240MHz or something else. The CORE generator has all its constraints set up for 200MHz. Note that this 200MHz has nothing to do with your fabric clock, as it comes from a different PLL on the FPGA. The yellow block does not interface to the dram itself, but simply FIFOs. When you write to the dram yellow block, you write to FIFOs that cross the clock domains. For streaming 144 bit data to DACs, you are reading 288 bit data from the dram at 1/2 your fabric clock rate. This gives lots of time for the FIFOs to read from the dram in little 8 word bursts and still deal with the dram refresh. I post this because we got confused by the documentation on the casper website. Also, we are moving from ROACH to ROACH2, and did not know the little differences in the dram. Also, it is natural to assume that the yellow block talks directly to the xilinx ddr3 controller and the ports correspond to the UI interface documented in the xilinx memory generator docs. This is not the case. The yellow block reads and writes fifos, and the ddr stuff is all hidden. The dram is a nice design by the way. I am just pointing out some details to save other folks some time. Tim Madden Argonne Lab
Re: [casper] Receive path automatic link training on ROACH2 10GbE
Thanks, Robin! This should do it. John Hi John, There is a few parts to this: The first thing is that the phy programming infrastructure is built into the new tcpborphserver with the ?phyprog katcp command. This version of tcpborphserver is contained in the romfs image. This can be extracted by mounting the romfs image as a loop device and then copying it out of /sbin $ mount -o loop roach2-root-phyprog-release-2015-04-01.romfs /mnt $ cd /mnt/sbin You can then run this version of tcpborphserver. Secondly, you will need the FPGA gateware which can also be found in the romfs in /boffiles. It's called r2_switch_tst_sys2_2014_May_20_0847.bof.gz You should unzip it and place it in the directory which tcpborphserver points to for bof files (default is /boffiles). The phy firmware binary obtained from Vitesse should also be placed in this directory. At this point, you could manually load the firmware by connecting to the new tcpborphserver via telnet. You should load the FPGA gateware with the ?progdev command and thereafter use the ?phyprog [mezz] [phy] command. Run this for each PHY on each mezzanine card, i.e ?progdev r2_switch_tst_sys2_2014_May_20_0847.bof ?phyprog 0 0 ?phyprog 0 1 ?phyprog 1 0 ?phyprog 1 1 You can use the ?phywatch [mezz] [phy] command to verify that the firmware is running. Issuing two consecutive ?phywatch commands should indicate whether the watchdog counter is incrementing and therefore confirm that the firmware is running. Run ?progdev to unload the FPGA again. There is also a script in the romfs at /etc/rc.physetup which can be run on the ROACH2 to perform this phy firmware loading once tcpborphserver is running. I hope this information helps you to get it up and running! Regards, Robin. On Wed, Apr 15, 2015 at 4:00 PM, John Ford jf...@nrao.edu wrote: Hi Robin. We currently use an NFS mounted file system. Can you tell me what I need to upgrade to make this work? Thanks! John Hi All, A new romfs containing the functionality to program the Vitesse VCS8488 PHYs on the ROACH2 10GbE mezzanine cards is available on Github at the following link: https://github.com/ska-sa/roach2_nfs_uboot/tree/master/boot and is called: roach2-root-phyprog-release-2015-04-01.romfs It contains an updated version of tcpborphserver, which includes the PHY programming commands, as well as scripts to load the PHY firmware at startup. As mentioned by Steve, the firmware is under NDA and can be obtained directly from Vitesse. Once the new romfs has been loaded via uboot, the following should be done for the upgrade to take effect (this only needs to be done once): - delete /var/.keep file. [*WARNING: this will delete all your previous /etc configurations*] - copy the vsc848x_EDC_FW_1_14.bin firmware file obtained from Vitesse into the /boffiles directory - reboot the ROACH2 On each subsequent reboot, the firmware will be loaded into each PHY's volatile memory. This process will also briefly program/deprogram the FPGA at startup and should complete within a minute or two. Alternatively, the PHYs can also be programmed manually by connecting to tcpborphserver and running the phyprog command. Regards, Robin van Wyk Software Engineer SKA-SA On Wed, Apr 15, 2015 at 1:39 PM, Stephen Dennehy sdenn...@ska.ac.za wrote: Hi All The Vitesse VCS8488 PHYs used on the 10GbE mezzanine cards on ROACH2 run, by default, with no Electronic Dispersion Compensation (EDC). They do however have provision for this to be enabled, by loading an application onto the PHY itself. We have tested this and found a dramatic decrease in receive path bit error rate. That's the good news. The bad news is that the application is under NDA from Vitesse, and can be accessed only after each user organisation signs a separate NDA directly with Vitesse. We asked if we could use one NDA for the whole CASPER community, but it was politely declined. The process is quite simple: - Register as a user on www.vitesse.com - Fill in the attached NDA form and send it to susan...@vitesse.com - Once approved, they will grant you access to the magical file vsc848x_EDC_FW_1_14.bin - Copy that file into the same directory as the bof files. - The latest version of romfs will look for the file at startup. If present and not yet loaded, it will load the application into volatile memory on the PHYs. If the file is not present, it will simply skip the loading and continue as before. The improvement in performance is significant, so it would probably be worth your while to do this, even though the NDA process is a bit cumbersome. *Loading instructions will follow shortly..* Thank you -- Steve Dennehy CBF Systems Engineer SKA-SA
