On Fri, 28 Aug 2009, John Ford wrote:
Hi all. We faced this problem with our GUPPI machine, and Randy
McCullough etal came up with a scheme to synchronize the 4 streams from 2
ibobs. It is indeed the case that the streams do not naturally remain
synchronized. If you use the OOB signals, extra transfers are inserted
into the data stream. Etc.
I will get Jason Ray or Randy to send you our synchronizer model file,
which is implemented in an extra FPGA on the BEE2.
We fought this problem for many months. It is indeed real...
Agreed!
Randy & I are both out of the office today, so we'll reply to
this again on Monday with more details.
Jason
We need a packet-enabled PFB and FFT, and then we could discard this
synchronizer business, but until then, we are stuck using a synchronous
system. We are running at 1.6 Gs/s.
John
Hi Jason,
I agree you could be fooled if, by chance, the ibob counters happen to be
synced. But, I think the likelihood of such an occurrence is low enough
that it probably wouldn't fool you very often or for very long. If the 1
pps sync *didn't* work, the values in the snap blocks would almost
certainly
be misaligned by many more than 1 or 2 counts.
But, I think this depends on how your sync generator is setup (whether the
counter feeding the snaps will free run upon programming, prior to setting
a
sync pulse period). In this case, the counter will only be regularly
reset
upon applying a period, and is thus a somewhat random function of the time
you enter a value into a sw register - thus making the likelihood of
synced
counters w/out a 1pps sync unlikely.
Triggering on the 1pps is indeed a more robust and specific test.
- Andrew
On 8/28/09 2:58 AM, "Jason Manley" <jasonman...@gmail.com> wrote:
Yeah, but what you're wanting to test here is if they are indeed
aligned. Agreed that the test using internal syncs would work if they
did trigger on a 1PPS, but then there's not much point in checking. So
you want to check if they did properly trigger on a 1PPS and if the
ibobs are correctly sync'd.
The problem with internal sync triggering
=========================================
So let's imagine that they don't properly sync on a 1PPS. Then their
onboard counters were not reset at the same time and are now running
in sync, but are misaligned.
Now let's say you are triggering your snap blocks off an internal
sync, generated from a local counter. You might trigger the snap
blocks on all the IBOBs and have themy wait for their next sync pulses
(which is still aligned on each ibob with their respective counter
values) before capturing a bunch of data (presumably a counter),
Depending on your internal sync period (let's imagine a sync period of
1second ~2^27), each snap block could contain the same counter values,
making it look like they're in sync, though in fact they might be out
by as much as a second, because the triggers could occur any time
within a second of the software arm. I have seen this happen with low
amplitude 1PPS inputs.
Debugging IBOB sync
===================
If you trigger off the external 1PPS, then you are assured of absolute
timing, rather than the internal sync, which may or may not be aligned
(it is precisely this that you need to test). Bearing in mind that the
exact trigger point of the 1PPS on each IBOB may jitter by up to 1
clock period (hence your snap's counter values might be misaligned by
up to 2 values). I have seen this with low amplitude or slewed (poor
distribution network) 1PPS pulses.
Visual check good
=================
Also, if you output the sync pulse of each ibob to an LED, you can
usually see pretty quickly if they are reasonably closely sync'd or
not. While not conclusive, it provides a nice sanity check.
A deployable check for sync
===========================
A nice way of building-in a check for sync in the deployed system is
to have the external 1PPS enable a register of your sync generator's
current counter. With a reasonably slow sync (I like 2^27 ~ 1 second),
you read the registered values from all the IBOBs at the same time.
Then you compare them in software. They should all be within 2 counts
of each other.
Jason
On 28 Aug 2009, at 11:35, Andrew Siemion wrote:
Hi Jason,
If the ibobs are clocked synchronously (as in Richard's setup), how
could
the internal syncs be out of alignment (after proper triggering on a
1pps?).
As I understand it, once you trigger on the 1pps the two ibob
internal syncs
should be perfectly aligned, both in cycle number and time (subject
to some
possible glitch)?
- Andrew
On 8/28/09 2:23 AM, "Jason Manley" <jasonman...@gmail.com> wrote:
Placing snap blocks on separate ibobs will not be a definitive test,
but would certainly be helpful. You would have to trigger them from
the external 1PPS, which is not guaranteed to be perfectly aligned
across boards after the first arm-trigger-sync operation (you may
jitter by one or two clocks). Bear this in mind when interpreting
the
two captures... that the values could be out by up to 2 counts.
Triggering the snap blocks off the internally-generated sync pulses
would defeat the purpose entirely, since the syncs would force the
local counters to trigger the snap blocks at the same values even if
they don't occur at the same time.
Jason
On 28 Aug 2009, at 11:10, Andrew Siemion wrote:
Hi Richard,
Did you add snap blocks to the counters prior to the xaui links,
triggered on a sync, to ensure that the ibobs are indeed
synchronized?
- Andrew
On 8/28/09 2:02 AM, "Richard Armstrong"
<armstrong.rich...@gmail.com> wrote:
Dear CASPERites,
I've been trying for some time to synchronise two XAUI streams from
two separate IBOB boards in a BEE2 board. I've not met with much
success. I'd be really grateful any ideas or help from anyone who
may have experience with this, or perhaps a better general
understanding of the issues I'm encountering.
The test system I've got is as follows: I'm clocking my IBOBs
synchronously and aligning them with a 1PPS input. Each IBOB has a
31-bit counter which is reset off a 'sync' pulse. The sync pulse is
generated by a circuit that is aligned with the 1PPS signal. I'm
sending the counter value multiplexed with the sync in a demux-by-2
XAUI block to the BEE2. At the BEE2, I'm testing the alignment of
the counters.
In my mind, as long as the ibob data rate is low enough, the XAUI
interface should provide complete clock-domain isolation. That is,
the bee2 should be able to be clocked at any rate, including a
lower rate than the ibob (provided, again that the data transfer
rate is low enough).
Worryingly, and unexpectedly, I find the counter values way out of
alignment, and perhaps worse than this, there seems to be jitter on
the difference between the counter values from cycle to cycle.
Andrew and Peter helped me out for a couple of days on their way to
Paris. We corrected some of the design errors (namely, selecting
demux-by-2 for 32-bit data), made sure the ibob clock was running
slower than the bee2 clock, and changed the length of the sync
pulse. However, we were unable to resolve the fundamental issues.
This week I've been trying to clock the BEE2 off an IBOB-generated
clock with an LVTTL-to-LVPECL clock driver. I changed the BEE2
design and changed a jumper to select user-clock on the BEE2 board.
Unfortunately, this doesn't seem to help much (although, I could be
doing it wrong) and I'm finding with this scheme that the BEE2 has
the tendancy to kill the running process after a few seconds to
minutes.
I know many of the more experienced CASPERites advise asynchronous
designs for this reason. However, I'm pretty sure someone else must
have disregarded this advice (or perhaps discovered this advice) by
pursuing XAUI synchronisation. Were these same issues seen? What's
the best clocking scheme in practice (fully synchronous or
downstream boards clcoked faster)? Does anyone have any ideas on
jitter?
Thanks,
Richard
