On 15 January 2014 16:28, Primiani, Rurik <rprimi...@cfa.harvard.edu> wrote:
> Hi Jack,
>
> Great work! The oversample mode is not something I realized existed!
Me neither -- I eventually stumbled across it in the SelectIO
Resources guide. (Where the order of the SERDES module outputs are
documented wrongly, just for extra fun)
>
> Previously when we've been able to get the tools to use HIGH bandwidth
> mode we could get away with just shifting the MMCM clock phase to
> remove glitches on the interface (i.e. no IODELAY adjustments). Have
> you tried just doing that?
I've run your calibration routine to look at the glitches vs. phase
values and it seems to find reasonably wide glitchless regions. But I
find the differences between the "slowest" and "fastest" bits from the
ADC to be around 4 IODELAY taps (312ps), so it seemed worth doing a
per-bit calibration anyway.
With IODELAY calibration only (without bothering with the MMCM phase)
so far I haven't had any glitches in 2 hours of running. Strangely,
when I ran the MMCM phasing calibration after IODELAY cal, things got
worse, but I've been wildly hacking software, so there's a very
reasonable chance I've broken something. I'll look into this after
I've let my current glitch counting test run for a bit longer.
>
> I'd like to merge your change into sma-wideband, could you perhaps
> send me a Github pull request?
Our git repos seem to have diverged quite a lot, but I've attached a
patch to peruse/edit/merge at your leisure. Obviously, YMMV :)
Cheers,
Jack
>
> Thanks!
> Rurik
>
> On Wed, Jan 15, 2014 at 7:51 AM, Jack Hickish <jackhick...@gmail.com> wrote:
>> Hi Ross,
>>
>> Thanks for the info -- I've actually got a 5 Gsps ROACH2 (demux 1:1)
>> pocket correlator compiled, and the final piece of the puzzle was
>> getting the ADCs running properly (I only got hold of the hardware
>> relatively recently). But it's always reassuring to know someone else
>> is using a similar system with success!
>>
>> For anyone that's interested, I think I may have solved my problem. I
>> changed the adc interface to use SERDES blocks in oversampling mode,
>> which allowed me to dispense with the high frequency MMCM output.
>> Without this troublesome output, the MMCM can be set into high
>> bandwidth mode and the data capture seems to perform much more
>> happily.
>> After this mod, and setting the IODELAY blocks appropriately, the ADC
>> appears to be capturing data happily without any problems....
>>
>> Cheers,
>>
>> Jack
>>
>> On 14 January 2014 18:18, Ross Williamson <rwilliam...@astro.caltech.edu>
>> wrote:
>>> Hi Jack,
>>>
>>> Not sure this helps as I'm using two DMUX 2:1 at 4GSPS as a correlator
>>> on a ROACH-1. The only think I would say is that I've put a fair
>>> amount of effort in planahead to try and get 5GSPS. I'm sure it could
>>> be done and I've thought I've got close a number of times but in the
>>> end I've just eaten that 500MHz of BW as too much effort for the
>>> little gain it gives in receiver noise. I would say that on a
>>> Roach-1, two boards at 5GSPS (4-bit) works fine if you are just
>>> dumping data to a snapshot. I believe a ROACH-2 is harder to reach
>>> timing.
>>>
>>> Ross
>>>
>>> On Tue, Jan 14, 2014 at 8:11 AM, Jack Hickish <jackhick...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> Like Weiwei, I'm trying to use the ADC5g at 5 Gsps. I've played with a
>>>> simple ADC to snap model, and (as Rurik warned) getting reliable data
>>>> capturing is difficult at this speed. I've tried per-bit calibration
>>>> of input data streams via IODELAYs in conjunction with phase-shifting
>>>> of the sampling clock, but can't seem to achieve glitchless data
>>>> capture for any reasonable length of time.
>>>>
>>>> In the email I'm replying to, Rurik suggests a number of ways around
>>>> this problem, but before I opt for forcing an unsupported MMCM mode by
>>>> compiling and running at different speeds, I thought a final desperate
>>>> plea for any advice on the maillist might be worthwhile. If anyone has
>>>> any words of wisdom, I'd be excessively grateful to hear from them.
>>>>
>>>> Cheers,
>>>>
>>>> Jack
>>>>
>>>> On 6 November 2013 16:31, Primiani, Rurik <rprimi...@cfa.harvard.edu>
>>>> wrote:
>>>>> Hi Weiwei,
>>>>>
>>>>> Generally the sma-wideband/mlib_devel has the most recent changes to
>>>>> the ADC5g yellow block, please use the latest commit of that repo (at
>>>>> this moment it is 2c80cb2).
>>>>>
>>>>> The problem your are seeing is related to a change that was introduced
>>>>> in the Xilinx tools (I believe from version 11 to 12) that increased
>>>>> the minimum frequency of the Virtex-6 MMCM's PFD to 135 MHz from 125
>>>>> MHz in HIGH bandwidth mode. This made it no longer possible to clock
>>>>> the fabric from an ADC5g running at >4800 Msps using an MMCM in HIGH
>>>>> bandwidth mode.
>>>>>
>>>>> There are a few options for moving forward (ordered by ease of
>>>>> implementation):
>>>>>
>>>>> 1. Clock the ADC below (and not including) 2400 MHz
>>>>> 2. Change MMCM from HIGH to LOW bandwidth mode
>>>>> 3. Compile your design with the ADC clocked above (and not including)
>>>>> 5400 MHz, but actually run it at 5 GHz
>>>>> 4. Force the Xilinx tools to allow the PFD to run at 125 MHz in HIGH
>>>>> bandwidth mode
>>>>> 5. Compile your design using the Xilinx 11 tools
>>>>>
>>>>> If you don't need to sample at exactly 5 Gsps then option 1 is the
>>>>> optimal solution. If 5 Gsps is critical then you can go with options 2
>>>>> or 3. Option 2 has non-ideal clock-to-out timing and may cause
>>>>> problems when trying to calibrate out jitter due to the data
>>>>> capturing. Option 3 may cause timing problems since your fabric clock
>>>>> will run at >337.5 MHz. I have not found a way to implement option 4
>>>>> but I can't see why this shouldn't be possible. Finally option 5 may
>>>>> or may not be useful to you.
>>>>>
>>>>> The SMA wideband correlator runs its ADCs below 4800 Msps. However we
>>>>> have bitcodes that we use to characterize the ADC that run at 5 Gsps
>>>>> and which have low resource utilization. For what it's worth we used
>>>>> option 3 to get around this issue and this appears to work for us.
>>>>> Whatever reason Xilinx had for changing the minimum PFD frequency to
>>>>> 135 MHz does not seem to cause problems for us.
>>>>>
>>>>> Hope this helps,
>>>>> Rurik
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Nov 6, 2013 at 2:29 AM, Weiwei Sun <su...@uw.edu> wrote:
>>>>>> I tried the casper-astro, ska-sa, sma-wideband. None of them seemly
>>>>>> works.
>>>>>>
>>>>>> Weiwei
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 5, 2013 at 10:20 PM, Primiani, Rurik
>>>>>> <rprimi...@cfa.harvard.edu>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Which version of mlib_devel are you using?
>>>>>>>
>>>>>>> Rurik
>>>>>>>
>>>>>>> On Tue, Nov 5, 2013 at 11:16 PM, Weiwei Sun <su...@uw.edu> wrote:
>>>>>>> > Hi,
>>>>>>> >
>>>>>>> > I have trouble to compile the adc clock rate of asiaa_adc5g block at
>>>>>>> > 2500MHz.
>>>>>>> > Block parameter: two-channel, ZDOK0, demux 1:1 .
>>>>>>> > System: roach2, clock source:adc0_clk, clock rate: 312.5MHz)
>>>>>>> >
>>>>>>> > The error is as follows:
>>>>>>> >
>>>>>>> > Creating block object: xps_adc5g
>>>>>>> > Problem with block: lab_test_adv5/asiaa_adc5g1
>>>>>>> > : An optimum PLL solution is not available!
>>>>>>> > Backtrace 1: xps_adc5g:175
>>>>>>> > Backtrace 2: gen_xps_files:229
>>>>>>> > Backtrace 3: run_Callback:149
>>>>>>> > Backtrace 4: casper_xps:82
>>>>>>> > Backtrace 5:
>>>>>>> >
>>>>>>> > @(hObject,eventdata)casper_xps('run_Callback',hObject,eventdata,guidata(hObject)):0
>>>>>>> > Error using gen_xps_files (line 242)
>>>>>>> > Error found during Object creation.
>>>>>>> >
>>>>>>> > There are a bunch of frequency justification in the xps_adc5g.m that I
>>>>>>> > just
>>>>>>> > can't understand. Could some one give me some suggestions on the
>>>>>>> > error
>>>>>>> > or
>>>>>>> > the codes? Does it mean that the 5G adc can't run at 2.5GHz?
>>>>>>> >
>>>>>>> > Any help or suggestions are very appreciated!
>>>>>>> >
>>>>>>> > Weiwei
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Ross Williamson
>>> Research Scientist - Sub-mm Group
>>> California Institute of Technology
>>> 626-395-2647 (office)
>>> 312-504-3051 (Cell)
>>
diff --git a/xps_base/XPS_ROACH2_base/pcores/adc5g_dmux1_interface_v1_00_a/hdl/vhdl/adc5g_dmux1_interface.vhd b/xps_base/XPS_ROACH2_base/pcores/adc5g_dmux1_interface_v1_00_a/hdl/vhdl/adc5g_dmux1_interface.vhd
index 0603441..de4811f 100644
--- a/xps_base/XPS_ROACH2_base/pcores/adc5g_dmux1_interface_v1_00_a/hdl/vhdl/adc5g_dmux1_interface.vhd
+++ b/xps_base/XPS_ROACH2_base/pcores/adc5g_dmux1_interface_v1_00_a/hdl/vhdl/adc5g_dmux1_interface.vhd
@@ -118,9 +118,9 @@ architecture behavioral of adc5g_dmux1_interface is
signal mmcm_rst : std_logic;
-- ISERDES signals
- signal isd_clk : std_logic;
- signal isd_clkn : std_logic;
signal isd_clkdiv : std_logic;
+ signal isd_clkdivn : std_logic;
+ signal isd_clkdiv90 : std_logic;
signal isd_rst : std_logic;
signal isd0_rst0 : std_logic_vector(adc_bit_width-1 downto 0);
signal isd1_rst0 : std_logic_vector(adc_bit_width-1 downto 0);
@@ -359,7 +359,7 @@ begin
MMCM0: MMCM_ADV
generic map (
- BANDWIDTH => "HIGH",
+ BANDWIDTH => "OPTIMIZED",
CLKFBOUT_MULT_F => mmcm_m,
DIVCLK_DIVIDE => mmcm_d,
CLKFBOUT_PHASE => 0.0,
@@ -387,7 +387,7 @@ begin
CLKINSEL => '1',
CLKIN1 => adc_clk_div,
CLKIN2 => '0',
- CLKOUT0 => mmcm_clkout0,
+ CLKOUT0 => open,
CLKOUT1 => mmcm_clkout1,
CLKOUT2 => mmcm_clkout2,
CLKOUT3 => mmcm_clkout3,
@@ -411,9 +411,8 @@ begin
--CBUF2a: BUFG port map (i=> mmcm_clkfbout, o=> mmcm_clkfbin);
mmcm_clkfbin <= mmcm_clkfbout;
- CBUF2b: BUFG port map (i=> mmcm_clkout0, o=> isd_clk);
CBUF2c: BUFG port map (i=> mmcm_clkout1, o=> isd_clkdiv);
- CBUF2d: BUFG port map (i=> mmcm_clkout2, o=> ctrl_clk90_out);
+ CBUF2d: BUFG port map (i=> mmcm_clkout2, o=> isd_clkdiv90);
CBUF2e: BUFG port map (i=> mmcm_clkout3, o=> ctrl_clk180_out);
CBUF2f: BUFG port map (i=> mmcm_clkout4, o=> ctrl_clk270_out);
@@ -421,7 +420,8 @@ begin
sync <= adc_sync;
ctrl_clk_out <= isd_clkdiv;
- isd_clkn <= not isd_clk;
+ ctrl_clk90_out <= isd_clkdiv90;
+ isd_clkdivn <= not isd_clkdiv;
-- purpose: Decode the datain pin and tap value
-- type : sequential
@@ -614,124 +614,144 @@ begin
iserdesx : for i in adc_bit_width-1 downto 0 generate
- iserdes0 : ISERDES_NODELAY
+ iserdes0 : ISERDESE1
generic map (
DATA_RATE => "DDR",
DATA_WIDTH => 4,
- INTERFACE_TYPE=> "NETWORKING",
+ INTERFACE_TYPE=> "OVERSAMPLE",
SERDES_MODE => "MASTER",
+ IOBDELAY => "BOTH",
NUM_CE => 2
)
port map (
- Q1 => data0d_pre(i),
- Q2 => data0c_pre(i),
- Q3 => data0b_pre(i),
- Q4 => data0a_pre(i),
+ Q1 => data0a_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data0d_pre(i),
+ Q2 => data0c_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data0c_pre(i),
+ Q3 => data0b_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data0b_pre(i),
+ Q4 => data0d_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data0a_pre(i),
Q5 => open,
Q6 => open,
- SHIFTOUT1 => open,
- SHIFTOUT2 => open,
- BITSLIP => '0',
- CE1 => '1',
- CE2 => '1',
- CLK => isd_clk,
- CLKB => isd_clkn,
- CLKDIV => isd_clkdiv,
- D => data0_delay(i),
- OCLK => '0',
- RST => isd_rst,
- SHIFTIN1 => '0',
- SHIFTIN2 => '0'
+ SHIFTOUT1 => open,
+ SHIFTOUT2 => open,
+ BITSLIP => '0',
+ CE1 => '1',
+ CE2 => '1',
+ CLK => isd_clkdiv,
+ CLKB => isd_clkdivn,
+ OCLK => isd_clkdiv90,
+ CLKDIV => '0',
+ DYNCLKSEL => '0',
+ DYNCLKDIVSEL => '0',
+ SHIFTIN1 => '0',
+ SHIFTIN2 => '0',
+ RST => isd_rst,
+ D => '0',
+ DDLY => data0_delay(i),
+ OFB => '0'
);
- iserdes1 : ISERDES_NODELAY
+ iserdes1 : ISERDESE1
generic map (
DATA_RATE => "DDR",
DATA_WIDTH => 4,
- INTERFACE_TYPE=> "NETWORKING",
+ INTERFACE_TYPE=> "OVERSAMPLE",
SERDES_MODE => "MASTER",
+ IOBDELAY => "BOTH",
NUM_CE => 2
)
port map (
- Q1 => data1d_pre(i),
- Q2 => data1c_pre(i),
- Q3 => data1b_pre(i),
- Q4 => data1a_pre(i),
+ Q1 => data1a_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data1d_pre(i),
+ Q2 => data1c_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data1c_pre(i),
+ Q3 => data1b_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data1b_pre(i),
+ Q4 => data1d_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data1a_pre(i),
Q5 => open,
Q6 => open,
- SHIFTOUT1 => open,
- SHIFTOUT2 => open,
- BITSLIP => '0',
- CE1 => '1',
- CE2 => '1',
- CLK => isd_clk,
- CLKB => isd_clkn,
- CLKDIV => isd_clkdiv,
- D => data1_delay(i),
- OCLK => '0',
- RST => isd_rst,
- SHIFTIN1 => '0',
- SHIFTIN2 => '0'
+ SHIFTOUT1 => open,
+ SHIFTOUT2 => open,
+ BITSLIP => '0',
+ CE1 => '1',
+ CE2 => '1',
+ CLK => isd_clkdiv,
+ CLKB => isd_clkdivn,
+ OCLK => isd_clkdiv90,
+ CLKDIV => '0',
+ DYNCLKSEL => '0',
+ DYNCLKDIVSEL => '0',
+ SHIFTIN1 => '0',
+ SHIFTIN2 => '0',
+ RST => isd_rst,
+ D => '0',
+ DDLY => data1_delay(i),
+ OFB => '0'
);
- iserdes2 : ISERDES_NODELAY
+ iserdes2 : ISERDESE1
generic map (
DATA_RATE => "DDR",
DATA_WIDTH => 4,
- INTERFACE_TYPE=> "NETWORKING",
+ INTERFACE_TYPE=> "OVERSAMPLE",
SERDES_MODE => "MASTER",
+ IOBDELAY => "BOTH",
NUM_CE => 2
)
port map (
- Q1 => data2d_pre(i),
- Q2 => data2c_pre(i),
- Q3 => data2b_pre(i),
- Q4 => data2a_pre(i),
+ Q1 => data2a_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data2d_pre(i),
+ Q2 => data2c_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data2c_pre(i),
+ Q3 => data2b_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data2b_pre(i),
+ Q4 => data2d_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data2a_pre(i),
Q5 => open,
Q6 => open,
- SHIFTOUT1 => open,
- SHIFTOUT2 => open,
- BITSLIP => '0',
- CE1 => '1',
- CE2 => '1',
- CLK => isd_clk,
- CLKB => isd_clkn,
- CLKDIV => isd_clkdiv,
- D => data2_delay(i),
- OCLK => '0',
- RST => isd_rst,
- SHIFTIN1 => '0',
- SHIFTIN2 => '0'
+ SHIFTOUT1 => open,
+ SHIFTOUT2 => open,
+ BITSLIP => '0',
+ CE1 => '1',
+ CE2 => '1',
+ CLK => isd_clkdiv,
+ CLKB => isd_clkdivn,
+ OCLK => isd_clkdiv90,
+ CLKDIV => '0',
+ DYNCLKSEL => '0',
+ DYNCLKDIVSEL => '0',
+ SHIFTIN1 => '0',
+ SHIFTIN2 => '0',
+ RST => isd_rst,
+ D => '0',
+ DDLY => data2_delay(i),
+ OFB => '0'
);
- iserdes3 : ISERDES_NODELAY
+ iserdes3 : ISERDESE1
generic map (
DATA_RATE => "DDR",
DATA_WIDTH => 4,
- INTERFACE_TYPE=> "NETWORKING",
+ INTERFACE_TYPE=> "OVERSAMPLE",
SERDES_MODE => "MASTER",
+ IOBDELAY => "BOTH",
NUM_CE => 2
)
port map (
- Q1 => data3d_pre(i),
- Q2 => data3c_pre(i),
- Q3 => data3b_pre(i),
- Q4 => data3a_pre(i),
+ Q1 => data3a_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data3d_pre(i),
+ Q2 => data3c_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data3c_pre(i),
+ Q3 => data3b_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data3b_pre(i),
+ Q4 => data3d_pre(i), -- THESE ARE DIFFERENT TO THOSE USED IN NETWORK MODE data3a_pre(i),
Q5 => open,
Q6 => open,
- SHIFTOUT1 => open,
- SHIFTOUT2 => open,
- BITSLIP => '0',
- CE1 => '1',
- CE2 => '1',
- CLK => isd_clk,
- CLKB => isd_clkn,
- CLKDIV => isd_clkdiv,
- D => data3_delay(i),
- OCLK => '0',
- RST => isd_rst,
- SHIFTIN1 => '0',
- SHIFTIN2 => '0'
+ SHIFTOUT1 => open,
+ SHIFTOUT2 => open,
+ BITSLIP => '0',
+ CE1 => '1',
+ CE2 => '1',
+ CLK => isd_clkdiv,
+ CLKB => isd_clkdivn,
+ OCLK => isd_clkdiv90,
+ CLKDIV => '0',
+ DYNCLKSEL => '0',
+ DYNCLKDIVSEL => '0',
+ SHIFTIN1 => '0',
+ SHIFTIN2 => '0',
+ RST => isd_rst,
+ D => '0',
+ DDLY => data3_delay(i),
+ OFB => '0'
);
end generate iserdesx;
diff --git a/xps_library/@xps_adc5g/xps_adc5g.m b/xps_library/@xps_adc5g/xps_adc5g.m
index 1e08ac8..5ef16bc 100644
--- a/xps_library/@xps_adc5g/xps_adc5g.m
+++ b/xps_library/@xps_adc5g/xps_adc5g.m
@@ -85,6 +85,7 @@ switch s.hw_sys
% to determine the M and D values for the PLL
% All swictching values are for V5 -1 speed grade
case 'ROACH'
+ oversample_factor = 1; %See ROACH2 case below for explanation
f_pfdmax = 449; % MHz
f_pfdmin = 19; % MHz
f_vcomax = 1000; % MHz
@@ -96,6 +97,15 @@ switch s.hw_sys
o0_allowed = 1:o0_prec:128;
o1_allowed = 1:128;
case 'ROACH2'
+ % We use a 2x oversampling SERDES module to allow data to be captured with half clock rate. This allows the MMCM to be configured in
+ % high bandwidth mode with an ADC sampling rate of 5 GHz +/- 200 MHz. Previously, this represented an edge case where there are no
+ % valid MMCM parameters unless bandwidth was set to LOW, which caused glitching on data capture.
+ % Only the ROACH2 demux 1:1 firmware has been altered in this was (I think the other modes should be fine without oversampling...(?))
+ if strcmp(demux, '1:1')
+ oversample_factor = 2;
+ else
+ oversample_factor = 1;
+ end
f_pfdmax = 450; % MHz, for bandwidth set to HIGH
f_pfdmin = 135; % MHz, for bandwidth set to HIGH
f_vcomax = 1200; % MHz
@@ -125,7 +135,7 @@ for r=r_allowed
pfd_freq = pll_freq/d;
for m=m_min:m_max
vco_freq = pfd_freq*m;
- o0 = vco_freq/s.adc_clk_rate;
+ o0 = vco_freq/s.adc_clk_rate * oversample_factor;
o1 = vco_freq/s.sysclk_rate;
% disp(sprintf('%d %d %d %f %f %f %f', r, d, m, pfd_freq, vco_freq, o0, o1));
if ~ismember(m, m_allowed)
