Glad to see some dialog on this, following it all. Ross, thanks for the very helpful and common sense suggestions. I think we know thermal isn't the trigger since we sometimes shut down and cool the system and then restart seeing the same issues. Measuring possible power supply droop is a good idea too, also npted. Not sure changing caps is practical because of disassembly etc, will keep it in mind. Thanks Jack, Dan, Dale too. Jonathan
On Tue, Sep 22, 2026 at 10:33 AM 'Ross Martin' via [email protected] <[email protected]> wrote: > Hi Jonathan, > > What things have you already tried to help narrow the problem down? > > Have you tried cooling the boards to see if that has an effect? (Or do > you have more issues in summer than winter?) > > Have you measured the power supply voltages going to the FPGA and to the > QDR? Perhaps some power supply components are getting weak and the > voltages are low. > > Have you tried replacing the capacitors? A very common issue with a lot > of old electronics is that the filter capacitors degrade, especially if > they are electrolytic. This can cause noise on the power rails, leading to > problems like this. > > Since you say that replacing the QDRs is sometimes effective, these other > problems become less likely. But if timing is marginal it's possible > multiple things can affect it. it doesn't hurt to check these off. > > Regards, > > Ross > > > On 9/22/26 04:23, 'Weintroub, Jonathan' via [email protected] > wrote: > > Hi Dale, hi Jack, and relevant CASPERites, > > Dale, while I am sorry you are having trouble with EOVSA, I am > simultaneously happy you raised the topic of QDR on the CASPER list. > > At SMA we have 68 ROACH2s installed in the operational SWARM (16 of which > are "aspirational" but still relevant), and between lab system and spares, > own close to 100 units. Recognizing that ROACH2 is a decade old, and many > have moved on, as they should, to newer tech (to which we also aspire), for > better or worse, we still depend on it for our primary facility back end. > > We have more failures than we'd like, and most of them are of QDR. The > the failures are in two broad categories: Either a ROACH2 doesn't close the > QDR calibration during start up, in which case part of the system doesn't > initialize; OR during operation data corruption with characteristics we've > come to recognize (narrow spikes in the visibilities) is traceable to QDR > data corruption. > > We have dealt with this in two primary ways, either swapping in spares > (but inevitably we are running out); and for a time we had a contract with > Digicom to swap QDRs (which required dis-assembly of the ROACH2 box with 5 > GS/s ADCs, mezzanines, cables etc), re-assemble and retest. This had > partial success, but since Mo sold Digicom the arrangement has come to an > end for various reasons, including that they no longer have a test server, > the QDR chips being far past end-of-life, etc. > > To no longer bury the lede: it is my view that what is really underlying > these issues is calibration and application of QDRs which, in our case at > least, works "by and large" but is on the margin and goes outside for some > reason, including aging. I base this on recollection of back and forth > when ROACH2 wss first being brought up, and there is some record of this in > the mailing list archive. > > Our application includes a 286 MHz fabric clock, -2 speed grade FPGAs. > And I vaguely remember that at build time we can pick the separate QDR > clock (100 or 200 MHz?). > > It is my hope that those who were at the center of the QDR development > might be able to contribute some technical context to my handwaving history > so that we at SMA can dig into what might be going on. (We recently > assigned a FPGA engineer to do so.) > > Jonathan > > > > On Tue, Sep 22, 2026 at 6:32 AM Jack Hickish <[email protected]> > wrote: > >> Hi Dale, >> >> As far as I'm aware there are no special settings needed for QDR - >> everything is baked into the firmware. Does the failure give any useful >> information? >> Has the new board passed its factory QDR tests at digicom? >> >> Cheers >> Jack >> >> On Mon, 21 Sept 2026 at 23:38, Dale Gary <[email protected]> wrote: >> >>> Hi All, >>> >>> We've been running our EOVSA correlator for nearly a decade now, but one >>> of our ROACH2 boards has finally failed. We replaced it with another >>> ROACH2 and installed the same KatADCs moved from the failed ROACH2. >>> However, the new ROACH2 board fails its QDR test on all three QDRs. Are >>> there some settings we have forgotten? Any other ideas? >>> >>> Thanks, >>> Dale Gary >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "[email protected]" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion visit >>> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAE1wuhZ%3DKb2n%2BDJ573t%2BaBj2yZb0ExKfGLn_BAvi4Mt4_u7uoA%40mail.gmail.com >>> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAE1wuhZ%3DKb2n%2BDJ573t%2BaBj2yZb0ExKfGLn_BAvi4Mt4_u7uoA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "[email protected]" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion visit >> https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSktCVbaUbUSnKAP%2BWfoFwzgzE45GuwUGu55dtR4OW%3DZMQ%40mail.gmail.com >> <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAG1GKSktCVbaUbUSnKAP%2BWfoFwzgzE45GuwUGu55dtR4OW%3DZMQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to the Google Groups > "[email protected]" <[email protected]> group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAPDrt26tig%3DEV7WMJS4G6X0GuOT4hD-JueC37mRHXbRcnHag3g%40mail.gmail.com > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAPDrt26tig%3DEV7WMJS4G6X0GuOT4hD-JueC37mRHXbRcnHag3g%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups " > [email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/9db7b019-b808-4a47-bbb2-046e6cae09d8%40ieee.org > <https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/9db7b019-b808-4a47-bbb2-046e6cae09d8%40ieee.org?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAPDrt25%3DrDUgogcxVzsBphVfrJZfO9TaoVgzPT4iDQbxNsjCjQ%40mail.gmail.com.
