Re: [casper] XAUI sync problem
I think what must be happening is even though your iBOB and BEE2 are running on the same clock, the XAUI links have independent 156.25 MHz clocks, so even though the same data rate is going in as coming out, the data is available at the other end of the link at slightly different times, so they drift in and out of sync. Although it seems like this would require two different clocks on the BEE2 end, so maybe this would be solved if you ensure that the pair of XAUI links you use is clocked from the same BREF oscillator. (I think that's 0 and 2 or 1 and 3 but I can't remember right now). Perhaps you can solve your problem as simply as not allowing reads from the XAUI until after say 32 clocks from when the iBOB begins transmitting, to allow the buffers to fill up a little. Glenn On 7/21/08, Jason Ray wrote: > All, > > As some of you may know, Randy & I have been working on getting reliable > data transmission across the inter-chip connections on the BEE2. We managed > to get that working reliably and have moved on to the next problem that has > reared its ugly head. > > Once we got back to the point of looking at spectra again, we soon noticed > that we are occasionally getting spurs, on the odd harmonics of our clock > freq +/- the input frequency. That is, if our input frequency is 51Mhz, > we'll see a spike at 51MHz, 149MHz, 251MHz, 549MHz, and 651MHz. Our iBOB is > using ADC_CLK which we have set to 800MHz, the iBOB provides the clock to > the BEE2 usr_clk2x input. > > Further investigation revealed that our two XAUI links (from the same iBOB) > are getting out of sync. The eight samples from the ADC are transmitted to > the BEE2 over XAUI... four samples on XAUI0 and the other four samples on > XAUI1. Once they arrive at the BEE2, they are sometimes out of sync. Its > hard to explain in an email, but if we look at the received sample values > (in BRAMs) we can discern the sine wave, but with discontinuities as if the > samples are scrambled. We have seen the sync & unsync condition come & go > over periods of a half hour or so, so maybe there's some sort of clock > drifting going on...? > > We verified the unsync condition by generating various test patterns in the > iBOB. If we transmit all 1's on one clock and all 0's on the next, the two > are out of sync at the BEE2. If we transmit all A's, then all 5's, the two > are generally in sync but we have seen it go out of sync, but not as often. > We also transmitted a 32-bit counter value across the two XAUIs and have > seen it go sync and unsync. > > Has anyone ever experienced this? Or are there any suggestions? I > attached a pdf of our iBOB sampler design. At the BEE2, we simply slice > apart the incoming XAUI data and shove it in BRAMs. > > Thanks in advance! > Jason > >
Re: [casper] XAUI sync problem
I think the 10GBE signal has clock embedded in it, the incoming clock should be used instead of local one. Very possibly this is already the case, but it's worth checking. Another guess is that there is a gounding problem and sometimes the common-ground voltage saturates the receivers. Easy to test by switching a CRT monitor on and off nearby. Setting up an oscilloscope to hunt for longer-than-possible ones or zeroes is a help, perhaps need to modify design to include a test point. I have seen both (although not in iBob environment), the latter is lot more common. Cheers, Jouko "Life is pretty simple: You do some stuff. Most fails. Some works. You do more of what works. If it works big, others quickly copy it. Then you do something else. The trick is to do something else." On Mon, 21 Jul 2008, G Jones wrote: I think what must be happening is even though your iBOB and BEE2 are running on the same clock, the XAUI links have independent 156.25 MHz clocks, so even though the same data rate is going in as coming out, the data is available at the other end of the link at slightly different times, so they drift in and out of sync. Although it seems like this would require two different clocks on the BEE2 end, so maybe this would be solved if you ensure that the pair of XAUI links you use is clocked from the same BREF oscillator. (I think that's 0 and 2 or 1 and 3 but I can't remember right now). Perhaps you can solve your problem as simply as not allowing reads from the XAUI until after say 32 clocks from when the iBOB begins transmitting, to allow the buffers to fill up a little. Glenn On 7/21/08, Jason Ray wrote: All, As some of you may know, Randy & I have been working on getting reliable data transmission across the inter-chip connections on the BEE2. We managed to get that working reliably and have moved on to the next problem that has reared its ugly head. Once we got back to the point of looking at spectra again, we soon noticed that we are occasionally getting spurs, on the odd harmonics of our clock freq +/- the input frequency. That is, if our input frequency is 51Mhz, we'll see a spike at 51MHz, 149MHz, 251MHz, 549MHz, and 651MHz. Our iBOB is using ADC_CLK which we have set to 800MHz, the iBOB provides the clock to the BEE2 usr_clk2x input. Further investigation revealed that our two XAUI links (from the same iBOB) are getting out of sync. The eight samples from the ADC are transmitted to the BEE2 over XAUI... four samples on XAUI0 and the other four samples on XAUI1. Once they arrive at the BEE2, they are sometimes out of sync. Its hard to explain in an email, but if we look at the received sample values (in BRAMs) we can discern the sine wave, but with discontinuities as if the samples are scrambled. We have seen the sync & unsync condition come & go over periods of a half hour or so, so maybe there's some sort of clock drifting going on...? We verified the unsync condition by generating various test patterns in the iBOB. If we transmit all 1's on one clock and all 0's on the next, the two are out of sync at the BEE2. If we transmit all A's, then all 5's, the two are generally in sync but we have seen it go out of sync, but not as often. We also transmitted a 32-bit counter value across the two XAUIs and have seen it go sync and unsync. Has anyone ever experienced this? Or are there any suggestions? I attached a pdf of our iBOB sampler design. At the BEE2, we simply slice apart the incoming XAUI data and shove it in BRAMs. Thanks in advance! Jason
Re: [casper] XAUI sync problem
hi jason, what cable lengths are you using? we recommend 1 meter max for ibob to bee2 cables. 3 meter cables have problems at high data rates. who made your cables? we've been using cables made by gore. what is the data rate you are sending over xaui links? eg: you are clocking the fpga at 200 MHz - do you send 64 bits every 5 nS, or 32 bits every 5 nS? do you load the xaui fifo on every clock or every other clock? in the packetized correlator, we generate a 1 PPS signal from a counter on the ibob (which is locked to external 1 PPS) and we send this signal with data over all xaui links. the bee2 uses the 1 PPS to make sure everything is phased up. it's very rare to loose sync with 1 meter cables. best wishes, dan G Jones wrote: I think what must be happening is even though your iBOB and BEE2 are running on the same clock, the XAUI links have independent 156.25 MHz clocks, so even though the same data rate is going in as coming out, the data is available at the other end of the link at slightly different times, so they drift in and out of sync. Although it seems like this would require two different clocks on the BEE2 end, so maybe this would be solved if you ensure that the pair of XAUI links you use is clocked from the same BREF oscillator. (I think that's 0 and 2 or 1 and 3 but I can't remember right now). Perhaps you can solve your problem as simply as not allowing reads from the XAUI until after say 32 clocks from when the iBOB begins transmitting, to allow the buffers to fill up a little. Glenn On 7/21/08, Jason Ray wrote: All, As some of you may know, Randy & I have been working on getting reliable data transmission across the inter-chip connections on the BEE2. We managed to get that working reliably and have moved on to the next problem that has reared its ugly head. Once we got back to the point of looking at spectra again, we soon noticed that we are occasionally getting spurs, on the odd harmonics of our clock freq +/- the input frequency. That is, if our input frequency is 51Mhz, we'll see a spike at 51MHz, 149MHz, 251MHz, 549MHz, and 651MHz. Our iBOB is using ADC_CLK which we have set to 800MHz, the iBOB provides the clock to the BEE2 usr_clk2x input. Further investigation revealed that our two XAUI links (from the same iBOB) are getting out of sync. The eight samples from the ADC are transmitted to the BEE2 over XAUI... four samples on XAUI0 and the other four samples on XAUI1. Once they arrive at the BEE2, they are sometimes out of sync. Its hard to explain in an email, but if we look at the received sample values (in BRAMs) we can discern the sine wave, but with discontinuities as if the samples are scrambled. We have seen the sync & unsync condition come & go over periods of a half hour or so, so maybe there's some sort of clock drifting going on...? We verified the unsync condition by generating various test patterns in the iBOB. If we transmit all 1's on one clock and all 0's on the next, the two are out of sync at the BEE2. If we transmit all A's, then all 5's, the two are generally in sync but we have seen it go out of sync, but not as often. We also transmitted a 32-bit counter value across the two XAUIs and have seen it go sync and unsync. Has anyone ever experienced this? Or are there any suggestions? I attached a pdf of our iBOB sampler design. At the BEE2, we simply slice apart the incoming XAUI data and shove it in BRAMs. Thanks in advance! Jason
Re: [casper] XAUI sync problem
All, Thanks a bunch for all the replies! Definitely some things to look into. See my responses to Dan's questions below. Jason At 11:29 AM 7/21/2008, Dan Werthimer wrote: hi jason, what cable lengths are you using? we recommend 1 meter max for ibob to bee2 cables. 3 meter cables have problems at high data rates. who made your cables? we've been using cables made by gore. For these links we are using 1m cables made by gore. what is the data rate you are sending over xaui links? eg: you are clocking the fpga at 200 MHz - do you send 64 bits every 5 nS, or 32 bits every 5 nS? do you load the xaui fifo on every clock or every other clock? Yes, the fpga is running at 200 MHz. We have the two XAUI links setup with demux=2 and we expect to transmit a new set of eight samples (two xaui links, each with four samples concat-ed into one 32-bit word) every clock. in the packetized correlator, we generate a 1 PPS signal from a counter on the ibob (which is locked to external 1 PPS) and we send this signal with data over all xaui links. the bee2 uses the 1 PPS to make sure everything is phased up. it's very rare to loose sync with 1 meter cables. best wishes, dan G Jones wrote: I think what must be happening is even though your iBOB and BEE2 are running on the same clock, the XAUI links have independent 156.25 MHz clocks, so even though the same data rate is going in as coming out, the data is available at the other end of the link at slightly different times, so they drift in and out of sync. Although it seems like this would require two different clocks on the BEE2 end, so maybe this would be solved if you ensure that the pair of XAUI links you use is clocked from the same BREF oscillator. (I think that's 0 and 2 or 1 and 3 but I can't remember right now). Perhaps you can solve your problem as simply as not allowing reads from the XAUI until after say 32 clocks from when the iBOB begins transmitting, to allow the buffers to fill up a little. Glenn On 7/21/08, Jason Ray wrote: All, As some of you may know, Randy & I have been working on getting reliable data transmission across the inter-chip connections on the BEE2. We managed to get that working reliably and have moved on to the next problem that has reared its ugly head. Once we got back to the point of looking at spectra again, we soon noticed that we are occasionally getting spurs, on the odd harmonics of our clock freq +/- the input frequency. That is, if our input frequency is 51Mhz, we'll see a spike at 51MHz, 149MHz, 251MHz, 549MHz, and 651MHz. Our iBOB is using ADC_CLK which we have set to 800MHz, the iBOB provides the clock to the BEE2 usr_clk2x input. Further investigation revealed that our two XAUI links (from the same iBOB) are getting out of sync. The eight samples from the ADC are transmitted to the BEE2 over XAUI... four samples on XAUI0 and the other four samples on XAUI1. Once they arrive at the BEE2, they are sometimes out of sync. Its hard to explain in an email, but if we look at the received sample values (in BRAMs) we can discern the sine wave, but with discontinuities as if the samples are scrambled. We have seen the sync & unsync condition come & go over periods of a half hour or so, so maybe there's some sort of clock drifting going on...? We verified the unsync condition by generating various test patterns in the iBOB. If we transmit all 1's on one clock and all 0's on the next, the two are out of sync at the BEE2. If we transmit all A's, then all 5's, the two are generally in sync but we have seen it go out of sync, but not as often. We also transmitted a 32-bit counter value across the two XAUIs and have seen it go sync and unsync. Has anyone ever experienced this? Or are there any suggestions? I attached a pdf of our iBOB sampler design. At the BEE2, we simply slice apart the incoming XAUI data and shove it in BRAMs. Thanks in advance! Jason
