Re: [casper] Latency in PFB and FFT
Hi Ross The guys at Berkeley and Ryan would probably have more detailed advice (especially regarding hand-placement) but the following are some general guidelines if trying to optimise timing from within System Generator; 1. Adding latency to an operation in System Generator results in the following; a. Register stages are added to sections within the operation so as to pipeline things and allow higher speed operation. This is useful where pipeline registers exist in cores e.g the DSP48 multiplier core. It is also useful in operations that can be pipelined e.g the cast/convert block uses a sequence of operations. b. Once all possible register stages within the operation are exhausted, the remaining latency is allocated after the operation. This latency will be limited in the benefit it adds as it is normally implemented in a single slice (the look up table can act as a shift register in Xilinx FGPAs and the final register stage of latency is implemented using the register in the slice). This leads to the following tips; 1. Avoid long chains of asynchronous logic. Add latency to operations involving large fanout or fanin e.g muxes, adders, comparators, cast/convert. Do a bit of thinking and research on how the various operations would be implemented under-the-hood. 2. Register inputs and outputs of blocks. 3. Use the CASPER Delays/pipeline block instead of the System Generator delay block on critical timing paths (the pipeline block forces the latency to be implemented in a pipeline of register stages instead of being absorbed into a single slice). 4. BRAMs followed by Multipliers often cause timing problems as they are limited and location constrained. These occur in the pfb_fir and fft. Add lots of BRAM latency to help (at least 3 to start) and Multiplier latency (4 would be a start but adding too much adds register stages *after* the Multiplier which is pointless). 5. Any input/output to/from yellow blocks (especially FPGA pins) should contain a pipeline block with a latency of at least 2 (allowing one register stage near source and one near destination). 6. Check to see if the System Generator block you are using has timing related options e.g the cast/convert has a 'Pipeline for maximum performance' option. Cheers Andrew Hi All, I'm trying to get the ROACH1 to run at 325MHz (5GSPS) in a simple spectrometer. This is obviously pushing the limits of the hardware - I'm kind or arbitrarily tweaking the latencies in these blocks to try and meet timing requirements. Are there any guidelines/notes I should be following - i.e should certain latencies match such as Add and bram where as say Fanout doesn't matter. Also are there any limits on these - i.e say 20 rather than 2? I'm sure if my understanding of the PFB and FFT was more than minimal this would be obvious... R -- Ross Williamson Research Scientist - Sub-mm Group California Institute of Technology 626-395-2647 (office) 312-504-3051 (Cell)
Re: [casper] Using Two ADC Boards in Sync
Hi, Ron, On May 6, 2013, at 2:28 PM, r...@physics.ucsb.edu wrote: > Having one ADC filling the FIFO with > its own clock would cause one signal to have a > delay relative to the other. Are you suggesting > the FIFO in the FPGA corrects this problem by > cancelling the delay? I don't know how/whether the ADC yellow blocks account for this (very shallow) ADC1 FIFO delay by delaying the ADC0 data accordingly. I have never thought about what portion of the measured relative delay was due to clock demux ambiguity vs unaccounted for clock-domain-crossing FIFO latency. When comparing the output of two ADCs, the relative delays need to be measured after every programming of an associated FPGA. Between programmings, the relative delay due to the clock demux ambiguity and clock-domain-crossing FIFO latency (if any) is very stable. Maybe you could get away with measuring it every power cycle on the ROACH/ROACH2 rather than every time you program the FPGA. I've only dealt with this on multi-iBob systems where the FPGA has a built-in PPC that boots every time the FPGA is reprogrammed. As I mentioned in earlier emails, the iBob startup code would reset (both!) the ADCs until they came up in sync with each other. This was actually detrimental because resetting ADC0 made the FPGA clock go away and come back which caused clocking havoc. It also meant that the ADC clock delay ambiguity relative to other iBobs was not constant across reprogramming the FPGA even though it would have been had the ADCs not been reset by the startup code. I think it was a well intentioned "feature" that had some unintended side effects. We typically measure the relative delays by injecting correlated noise or observing a broadband calibrator (e.g. a quasar), correlating the inputs, and then fitting a phase slope across the cross correlation frequency spectrum. This also measures other sources of relative delay such as mismatched cable lengths. Dave
Re: [casper] Using Two ADC Boards in Sync
The reason I wanted the two ADC's in sync was to better resolve the phase difference between two signals at each frequency channel of the two FFT spectra. I determine the phase differences using the complex FFT outputs. In the past, the phase differences of the ADC units was too small to be a problem when studying signals at frequencies significantly lower than the ADC clock's frequency. Now there is reason to resolve the phase difference with finer resolution even if the signals are at lower frequencies. The ADC blocks have eight outputs. My suggestion that one ADC would be out of phase by multiples of 90 degrees should have said multiples of 45 degrees. Having one ADC filling the FIFO with its own clock would cause one signal to have a delay relative to the other. Are you suggesting the FIFO in the FPGA corrects this problem by cancelling the delay? > Hi, Ron, > > [I meant to send this to the list the first time.] > > On May 3, 2013, at 3:57 PM, r...@physics.ucsb.edu wrote: > >> I expect >> the ADC generating the FPGA clock will be in >> sync > > By definition! > >> but the other ADC may be 90, 180 or 270 >> degrees out of phase with the first ADC. > > Don't forget 0 degrees out of phase! :-) > > Even though the clocks can be out of phase, the data from each ADC is > clocked into the FPGA using that ADC's clock to ensure proper setup/hold. > Internal to the FPGA there is a cross-clock-domain FIFO that transfers the > data from ADC1 into the clock domain of ADC0 (which is the main fabric > clock domain), so all the data end up on one common clock domain. > >> Do the yellow blocks sync the ADC's on a ROACH? > > I don't know. Even if it does sync the two ADCs on one ROACH, it can't > sync the ADCs across two ROACHes, so systems that have with more than one > ROACH need to deal with the out-of-phase potential anyway. This is > usually done by measuring the difference using a common input signal (e.g. > noise or the sky). It is quite stable so it only needs to be done once. > > If you care about this, you will probably also care about line length > variations. However you plan on calibrating that can also be used to > calibrate the phase offset. The ADC clock phase offset is easier to > handle in some ways since it can only be one of several discrete values. > > Hope this helps, > Dave > >
[casper] Latency in PFB and FFT
Hi All, I'm trying to get the ROACH1 to run at 325MHz (5GSPS) in a simple spectrometer. This is obviously pushing the limits of the hardware - I'm kind or arbitrarily tweaking the latencies in these blocks to try and meet timing requirements. Are there any guidelines/notes I should be following - i.e should certain latencies match such as Add and bram where as say Fanout doesn't matter. Also are there any limits on these - i.e say 20 rather than 2? I'm sure if my understanding of the PFB and FFT was more than minimal this would be obvious... R -- Ross Williamson Research Scientist - Sub-mm Group California Institute of Technology 626-395-2647 (office) 312-504-3051 (Cell)
Re: [casper] Using Two ADC Boards in Sync
Hi, Dan, On May 6, 2013, at 9:23 AM, Dan Werthimer wrote: > on the yellow blocks i'm familiar with, > if the two adc boards can start up out of sync, > on power up (initialization), one of the adc's (the slave) > is reset as many times as needed until > the slave adc is in sync with master adc. > the fpga uses the clock from the master adc. TinyShell on the iBob used to have start-up software that would sync the ADCs' clocks (at least for the original ADC2x1000-8, aka iADC), but I am not aware of anything equivalent on the ROACH. Has this clock synchronization logic been moved into any ADC yellow block's HDL? Thanks, Dave
Re: [casper] Using Two ADC Boards in Sync
Hi, Ron, [I meant to send this to the list the first time.] On May 3, 2013, at 3:57 PM, r...@physics.ucsb.edu wrote: > I expect > the ADC generating the FPGA clock will be in > sync By definition! > but the other ADC may be 90, 180 or 270 > degrees out of phase with the first ADC. Don't forget 0 degrees out of phase! :-) Even though the clocks can be out of phase, the data from each ADC is clocked into the FPGA using that ADC's clock to ensure proper setup/hold. Internal to the FPGA there is a cross-clock-domain FIFO that transfers the data from ADC1 into the clock domain of ADC0 (which is the main fabric clock domain), so all the data end up on one common clock domain. > Do the yellow blocks sync the ADC's on a ROACH? I don't know. Even if it does sync the two ADCs on one ROACH, it can't sync the ADCs across two ROACHes, so systems that have with more than one ROACH need to deal with the out-of-phase potential anyway. This is usually done by measuring the difference using a common input signal (e.g. noise or the sky). It is quite stable so it only needs to be done once. If you care about this, you will probably also care about line length variations. However you plan on calibrating that can also be used to calibrate the phase offset. The ADC clock phase offset is easier to handle in some ways since it can only be one of several discrete values. Hope this helps, Dave
Re: [casper] Using Two ADC Boards in Sync
hi ron, in a two adc system, the fpga uses only one of the adc clocks. on the yellow blocks i'm familiar with, if the two adc boards can start up out of sync, on power up (initialization), one of the adc's (the slave) is reset as many times as needed until the slave adc is in sync with master adc. the fpga uses the clock from the master adc. dan On Fri, May 3, 2013 at 4:00 PM, wrote: > I am sorry. I mean the other ADC may be out > of phase with the FPGA clock. > > > No. I want to have both ADC boards be of the > > same type. The new concern I have is that you > > express doubt that the yellow blocks will > > sync the two ADC's on our ROACH board. If one > > ADC is out of sync with the other, I expect > > the ADC generating the FPGA clock will be in > > sync but the other ADC may be 90, 180 or 270 > > degrees out of phase with the first ADC. Do > > the yellow blocks sync the ADC's on a ROACH? > > > > I use a 2011 version of the CASPER library > > since upgrading has often caused the need to > > modify all my model designs. > > > >> > >> On May 3, 2013, at 12:09 PM, Dan Werthimer wrote: > >> > >>> the yellow block initialization checks the clock > >>> output from both adc's and resets one of > >>> the adc's until both clocks are synced up. > >> > >> I know this happened on the iBobs, but I'm not sure it is still the case > >> on the ROACH (or ROACH2). > >> > >> Ron, are you wanting to use two different kinds of ADC cards at the same > >> time on a single ROACH? > >> > >> Dave > >> > >> > >> > > > > > > > > >
