Hi Sam, I'm not familiar with the music board, but I wonder if it would benefit from tuning the clock vs data skew on the adc->fpga links to tune the data capture for reliability.
Are you coming to the CASPER workshop? If so I suggest bringing a music board and maybe somebody can donate some time to investigate your problem. Cheers Jack On Thu, Aug 3, 2017, 10:57 PM Sam Gordon <[email protected]> wrote: > Hi all, > > I seek your insight into an issue which my collaborators and I have been > troubleshooting for some time. It has to do with a 1024 channel Roach2 > spectrometer which uses the MUSIC DAC/ADC board. The Roach is clocked at > 256 MHz, which it derives from a 512 MHz input to the MUSIC board (taken > from one of the ADC channels). After making several duplicates of the > system (hardware, firmware and software), we noticed that a few of them > (something like one out of six) exhibit spurious voltage spikes in the ADC > time streams, which are visible via snap block readout. The effect of the > glitches is to increase the overall amplitude noise level at the channel > outputs, so we'd like to minimize or eliminate them if at all possible. > > Here is a rundown of observations and some attempts at troubleshooting so > far: > > Signs and symptoms: > - Voltage spikes in the digitized time streams of multitone frequency > combs. The glitches appear to be non-periodic, and vary in amplitude over > the full scale of the ADCs. They are present in both the I and Q channels, > but can be worse in a single channel. They are also observed, to a lesser > extent, when the I/Q inputs are terminated in 50 ohms. No glitches are > observed in the DAC output, which is looped directly back to the ADCs > during these measurements. The glitches only present in waveforms > containing more than a single frequency. > > - On an afflicted system, the glitches disappear when the clock is either > slowed to < ~430 MHz, or increased above ~530 MHz. On a system which is not > afflicted, the glitches can be induced by slowing the clock to below ~500 > MHz. > > - They have not been remedied by swapping MUSIC boards, although their > characteristics differ somewhat between different board combinations. When > a MUSIC board is taken from an afflicted system and used with a reliable > Roach, no glitches are observed. > > - Glitches are not observed in five systems which have had their FPGA fans > removed and replaced with heat pipe assemblies. However, one of these > systems exhibited glitches when the mounting bolts on the FPGA heat sink > were overtightened by a small amount. When they were loosened, the glitches > subsided immediately. > > Attempts at troubleshooting: > - Firmware: Identical firmware used on each of the systems. Given the > correlation with clock rate, timing is a likely culprit. Using MSSGE, the > firmware compiles with a timing score of zero. However, the best case > positive slack is quite small (a few picoseconds). I've attempted to > increase this by using pblocks in PlanAhead, as well as the get lucky > approach with SmartXplorer, without much success. The ADC MMCM is > configured for high bandwidth, in accordance with a previous post on MUSIC > board glitches. > > - Hardware: As mentioned above, the glitches haven't been observed when > running with heat pipes instead of fans. While watching a live stream of > the ADC snap readout on one of the afflicted systems, I unplugged the power > leads to the FPGA fan. The spikes gradually subsided over a few seconds. > When the fan was powered on again, they returned on the same time scale. > This effect was not reproducible on an identical system. > > - We've had no luck using different synthesizers as clock sources, > filtering, attenuating or otherwise isolating the clock line to the MUSIC > boards. > > Much appreciated, > Sam Gordon > Astrophysics PhD candidate > Arizona State University > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected].

