Re: [casper] Application of SMA correlator design to a larger array

[Gary, Dale E.]

Hi Clifford and All,

Thanks for your responses.  After looking into our design further, it looks
like we are moving toward a somewhat narrower bandwidth, which makes it
compatible with the Skarab ADC.  Since this is an entirely new platform for
me, I'll need some help visualizing the architecture needed for a 10
antenna (or 16, if 2^n is required) dual-pol system.  I assume that the
paradigm of a packet-switched correlator has not changed, so we are talking
about a set of F-engines feeding a set of X-engines.  With 3, 4-input ADCs,
we could get 6 dual-pol antennas into one Skarab, but are the FPGA
resources able to handle so many F-engines?  Can someone estimate for me
what the practical limit (number of inputs/Skarab) would be, for a generic
F-engine design?  For our ROACH-2-based EOVSA correlator, we actually have
both F- and X-engine functions on each ROACH-2, but what would be the
preferred architecture in the Skarab case?

To update the design concept from what I initially posted, we now want to
digitize roughly 1 - 1.2 GHz (depends on how broadband we can make the horn
feed) from 10 dual-pol antennas, channelizing to 4096 or more channels.
We'll need ~10 ms dump times for the science case, which for baselines for
10 antennas comes to about 360 MB/s, if I count correctly, but if we have
to dump baselines for all 16 inputs it becomes about 1 GB/s.  This is no
problem for a 40 Gbe system, I guess.

Could anyone venture a ballpark guess as to the cost of a Skarab-based
system to do this?

Many Thanks,
Dale

On Thu, Nov 9, 2017 at 5:21 AM, Clifford van Dyk 
wrote:

> Hi Dale
>
> In case you are interested in considering the Skarab platform, I have
> attached the latest Skarab product glossies to this email (there is also
> a lot if info on the Casper Skarab web pages). Currently, only one
> 4-channel, 3 GSPS 14-bit ADC mezzanine exists, that can (practically) be
> populated on any of the Skarab's 4 mezzanine sites. Since one site is
> dedicated to 4 x 40 Gb Ethernet, this has a practical limit of 12 ADC
> channels per Skarab. While the ADC is not capable of delivering a 2 GHz
> instantaneous bandwidth you are after (Nyquist is limited to 1.5 GHz),
> it has a number of other attractive features, like multiple built-in
> DDCs behind each ADC channel. This may offload some of the high speed
> trickery/resource that would otherwise go into the FPGA, opening up the
> resource for other use.
>
> Alternatively, in keeping with the direction that SKA-SA is heading
> w.r.t. external ADCs (Jason's comments below), the Skarab 4 x 40 GbE
> Mezzanine can actually run any high speed SERDES protocol between the
> QSFP+ connectors and the FPGA (there are no Ethernet-specific PHYs in
> between - PHY is fully implemented in the FPGA). So while SKA-SA prefers
> 40 GbE interfacing everywhere, it may be possible to use other SERDES
> protocols to attach directly to external ADCs (e.g. JESD, ESIStream)
> over the QSFP+ connectors - you can essentially see the mezzanine as
> capable of carrying up to 160 Gbps over 16 configurable SERDES lanes.
>
> Let me know if you have any specific questions around the platform.
>
> Kind regards,
> Clifford
>
> On 11/8/2017 9:24 AM, Jason Manley wrote:
> > Hi Dale
> >
> > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are
> all using) would scale simply to your needs. You could easily build it out
> of ROACH2s or SKARABs at your scales. It would be a bit more expensive than
> the SWARM-type solution, in that it'd need additional hardware because it's
> not as resource-efficient as the SWARM design (you're going to struggle to
> beat SWARM with any other design!). A ROACH2 packetised solution would
> allow a number of different ADC options, including the the same one that
> SWARM uses.
> >
> > SKA-SA is moving in a slightly different direction with ADCs for our
> current and future systems. We have moved the ADC off the digital
> processing boards in the interests of reducing noise, and increasing
> overall system performance (constrain analogue bits to the antennas, with
> digital backhaul and processing to improve linearity and dynamic range, for
> example). We are not alone in our thinking here, but many in the CASPER
> community have not caught on to the benefits of this yet. And,
> unfortunately, it is a more expensive solution overall. At the moment,
> there are no universal/community off-board ADCs. Everyone's got really
> expensive, custom-designed ones (a possible gap in the CASPER lineup,
> here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we
> haven't really designed any ADCs to mate directly with them. At the moment,
> there is actually one ADC that plugs directly into a SKARAB, but I'm not
> sure it'd be appropriate for your needs, and it has no yellow block yet
> (though Peralex promised to deliver one at the last CASPER workshop if a
> customer wanted it).
> >
> > Another consideration is that the newer 

Re: [casper] Application of SMA correlator design to a larger array

[Gary, Dale E.]

Hi Jason,

Thanks for the quick estimate.  If we are targeting 10 antennas, can't the
correlator be made to work with only 10 F engines?  The packets of the
missing F engines would just not arrive at the X-engine, and hence the
associated baselines would be zeroed or just ignored.  We might also design
the X-engine not to send blank data, and hence reduce the size of the
switch needed.  This might even allow us to halve the number of X-engine
boards, but I am not sure about that.

A 4-tap PFB is fine, and we will do fine delay correction off the boards
(phase correction in software), so only coarse delay steps are needed.
This is what we do for EOVSA, very successfully.  We will probably include
spectral kurtosis calculation, though, which means accumulating
power-squared.   Still, I think we have a shot at 1 F-engine board per
antenna.

Regards,
Dale

On Wed, Nov 8, 2017 at 2:24 AM, Jason Manley  wrote:

> Hi Dale
>
> The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are
> all using) would scale simply to your needs. You could easily build it out
> of ROACH2s or SKARABs at your scales. It would be a bit more expensive than
> the SWARM-type solution, in that it'd need additional hardware because it's
> not as resource-efficient as the SWARM design (you're going to struggle to
> beat SWARM with any other design!). A ROACH2 packetised solution would
> allow a number of different ADC options, including the the same one that
> SWARM uses.
>
> SKA-SA is moving in a slightly different direction with ADCs for our
> current and future systems. We have moved the ADC off the digital
> processing boards in the interests of reducing noise, and increasing
> overall system performance (constrain analogue bits to the antennas, with
> digital backhaul and processing to improve linearity and dynamic range, for
> example). We are not alone in our thinking here, but many in the CASPER
> community have not caught on to the benefits of this yet. And,
> unfortunately, it is a more expensive solution overall. At the moment,
> there are no universal/community off-board ADCs. Everyone's got really
> expensive, custom-designed ones (a possible gap in the CASPER lineup,
> here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we
> haven't really designed any ADCs to mate directly with them. At the moment,
> there is actually one ADC that plugs directly into a SKARAB, but I'm not
> sure it'd be appropriate for your needs, and it has no yellow block yet
> (though Peralex promised to deliver one at the last CASPER workshop if a
> customer wanted it).
>
> Another consideration is that the newer platforms (SKARAB, SNAP etc) are
> using the new JASPER toolflow, which is under active development and has
> support from the bigger CASPER developers. The older CASPER flow is still
> supported on ROACH2s, but new features and bug-fixes are not being
> back-ported and development there has stagnated. Since Xilinx will not
> support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be
> supported by all the new tools. I don't know of anyone using JASPER with
> ISE (though, it was possible at one time).
>
> If you wanted to build your correlator out of ROACH2s or SKARABs (they
> have similar processing capacities), a quick back-of-the-envelope,
> worst-case calculation suggests that, for a 16 dual-pol, 4k channel 8-tap,
> 2GHz BW correlator (I can almost hear the infomercial already: "act now,
> and we'll throw in a free beamformer"; you'd fit a beam or two in the spare
> capacity within the X-engine boards, FWIW), a packetised design would need
> something like:
>
> 32x F-engine boards (though I'd say there's a good chance you could
> squeeze it into 16 boards).
> 32x X-engine boards (30 if you don't want to process the band edges)
> 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port
> switch with some loopback trickery, which would be much cheaper).
>
> It looks like you will be BRAM limited in both cases (otherwise you could
> halve the board counts). You could also opt for some BRAM-saving tradeoffs
> to ease fitment of two polarisations onto an F-engine. For example you
> could drop down to a 4-tap PFB, or reduce the delay-correction resolution
> (MeerKAT's specs, upon which I based the numbers above, are overkill for
> most applications). If you're using a single network switch, you might also
> be able to reduce packet buffer requirements, which currently use BRAM, too.
>
> The beauty of this design is in its flexibility. You can access the raw,
> intermediate data streams on the switch, swap out portions of the design
> for computers/GPUs (eg LEDA and HERA), add more antennas incrementally,
> increase or decreased processed bandwidth, change spectral resolution etc.
> all with a quick parameter change and a recompile.
>
> Jason Manley
> Functional Manager: DSP
> SKA-SA
>
> Cell: +27 82 662 7726
> Work: +27 21 506 7300
>
> On 07 Nov 2017, at 20:02, Jonathan 

Re: [casper] Application of SMA correlator design to a larger array

[Jason Manley]

Hi Dale

The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are all 
using) would scale simply to your needs. You could easily build it out of 
ROACH2s or SKARABs at your scales. It would be a bit more expensive than the 
SWARM-type solution, in that it'd need additional hardware because it's not as 
resource-efficient as the SWARM design (you're going to struggle to beat SWARM 
with any other design!). A ROACH2 packetised solution would allow a number of 
different ADC options, including the the same one that SWARM uses.

SKA-SA is moving in a slightly different direction with ADCs for our current 
and future systems. We have moved the ADC off the digital processing boards in 
the interests of reducing noise, and increasing overall system performance 
(constrain analogue bits to the antennas, with digital backhaul and processing 
to improve linearity and dynamic range, for example). We are not alone in our 
thinking here, but many in the CASPER community have not caught on to the 
benefits of this yet. And, unfortunately, it is a more expensive solution 
overall. At the moment, there are no universal/community off-board ADCs. 
Everyone's got really expensive, custom-designed ones (a possible gap in the 
CASPER lineup, here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, 
and so we haven't really designed any ADCs to mate directly with them. At the 
moment, there is actually one ADC that plugs directly into a SKARAB, but I'm 
not sure it'd be appropriate for your needs, and it has no yellow block yet 
(though Peralex promised to deliver one at the last CASPER workshop if a 
customer wanted it).

Another consideration is that the newer platforms (SKARAB, SNAP etc) are using 
the new JASPER toolflow, which is under active development and has support from 
the bigger CASPER developers. The older CASPER flow is still supported on 
ROACH2s, but new features and bug-fixes are not being back-ported and 
development there has stagnated. Since Xilinx will not support Virtex 6 in 
Vivado, ROACH2 (and earlier boards) can never be supported by all the new 
tools. I don't know of anyone using JASPER with ISE (though, it was possible at 
one time).

If you wanted to build your correlator out of ROACH2s or SKARABs (they have 
similar processing capacities), a quick back-of-the-envelope, worst-case 
calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, 2GHz BW 
correlator (I can almost hear the infomercial already: "act now, and we'll 
throw in a free beamformer"; you'd fit a beam or two in the spare capacity 
within the X-engine boards, FWIW), a packetised design would need something 
like:

32x F-engine boards (though I'd say there's a good chance you could squeeze it 
into 16 boards).
32x X-engine boards (30 if you don't want to process the band edges)
1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port switch 
with some loopback trickery, which would be much cheaper).

It looks like you will be BRAM limited in both cases (otherwise you could halve 
the board counts). You could also opt for some BRAM-saving tradeoffs to ease 
fitment of two polarisations onto an F-engine. For example you could drop down 
to a 4-tap PFB, or reduce the delay-correction resolution (MeerKAT's specs, 
upon which I based the numbers above, are overkill for most applications). If 
you're using a single network switch, you might also be able to reduce packet 
buffer requirements, which currently use BRAM, too.

The beauty of this design is in its flexibility. You can access the raw, 
intermediate data streams on the switch, swap out portions of the design for 
computers/GPUs (eg LEDA and HERA), add more antennas incrementally, increase or 
decreased processed bandwidth, change spectral resolution etc. all with a quick 
parameter change and a recompile.

Jason Manley
Functional Manager: DSP
SKA-SA

Cell: +27 82 662 7726
Work: +27 21 506 7300

On 07 Nov 2017, at 20:02, Jonathan Weintroub  wrote:

> Hi Dale,
> 
> I’ll offer a few bullets on SWARM, the new SMA system.
> 
> 1.  SWARM is all open source and shared via CASPER and you are welcome to use 
> it as is, or develop it further to adapt it to a new application, indeed it 
> would be very pleasing to see the design used in some other instrument.
> 
> 2.  There is a paper which is worth reading to understand what SWARM is and 
> what it does.  Take a careful look if you are contemplating using the design.
> http://www.worldscientific.com/doi/pdf/10.1142/S2251171716410063
> You can get insight the paper without looking at the gory details of source 
> codes, both bitcodes and associated software, but if you want to dig even 
> deeper, sources are all shared here:
> http://www.github.com/sma-wideband.
> 
> 3.  You are correct SWARM processes 2 GHz blocks of *usable” bandwidth.  The 
> Nyquist band is somewhat wider, 2.288 GHz.   That Nyquist band is divided 
> into 16,384 channels (not 1024), so in fact it 

Re: [casper] Application of SMA correlator design to a larger array

[Dan Werthimer]

hi dale,

i'd be happy to talk to you on the phone about pros and cons of different
options if you'd like.
jack also told me a bit about your plans - he'd be good at providiing
advise on the different options..

dan

510-418-0546


On Tue, Nov 7, 2017 at 9:29 AM, Gary, Dale E.  wrote:

> Dear Jonathan (and the rest of the CASPER list, in case anyone has
> additional comments),
>
> I am looking into a new project that would require processing around 2 GHz
> of bandwidth on of-order 10 (but more than 8) dual-polarization antennas.
> Our science case calls for at least 4096 frequency channels.  My
> understanding is that the SMA correlator design is for a similar bandwidth,
> for 8 dual-pol antennas, but 1024 channels or something similar. We do not
> want to spend a lot of resources on correlator design, so my question is
> whether it is possible and would it make sense to adapt the SMA design to a
> 16-antenna, dual-pol, 4096-channel system, or whether it is better (or
> necessary) to leave the ROACH-2 designs behind and move to one of the newer
> platforms?  If the latter, what digitizer bandwidths are available, and
> which board (SNAP2, Scarab, others?) would be most appropriate to a new
> project of this scope?
>
> Thanks,
> Dale
>
> --
> You received this message because you are subscribed to the Google Groups "
> casper@lists.berkeley.edu" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to casper+unsubscr...@lists.berkeley.edu.
> To post to this group, send email to casper@lists.berkeley.edu.
>

-- 
You received this message because you are subscribed to the Google Groups 
"casper@lists.berkeley.edu" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to casper+unsubscr...@lists.berkeley.edu.
To post to this group, send email to casper@lists.berkeley.edu.