[casper] Application of SMA correlator design to a larger array
Dear Jonathan (and the rest of the CASPER list, in case anyone has additional comments), I am looking into a new project that would require processing around 2 GHz of bandwidth on of-order 10 (but more than 8) dual-polarization antennas. Our science case calls for at least 4096 frequency channels. My understanding is that the SMA correlator design is for a similar bandwidth, for 8 dual-pol antennas, but 1024 channels or something similar. We do not want to spend a lot of resources on correlator design, so my question is whether it is possible and would it make sense to adapt the SMA design to a 16-antenna, dual-pol, 4096-channel system, or whether it is better (or necessary) to leave the ROACH-2 designs behind and move to one of the newer platforms? If the latter, what digitizer bandwidths are available, and which board (SNAP2, Scarab, others?) would be most appropriate to a new project of this scope? Thanks, Dale -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To post to this group, send email to casper@lists.berkeley.edu.
Re: [casper] Application of SMA correlator design to a larger array
hi dale, i'd be happy to talk to you on the phone about pros and cons of different options if you'd like. jack also told me a bit about your plans - he'd be good at providiing advise on the different options.. dan 510-418-0546 On Tue, Nov 7, 2017 at 9:29 AM, Gary, Dale E. wrote: > Dear Jonathan (and the rest of the CASPER list, in case anyone has > additional comments), > > I am looking into a new project that would require processing around 2 GHz > of bandwidth on of-order 10 (but more than 8) dual-polarization antennas. > Our science case calls for at least 4096 frequency channels. My > understanding is that the SMA correlator design is for a similar bandwidth, > for 8 dual-pol antennas, but 1024 channels or something similar. We do not > want to spend a lot of resources on correlator design, so my question is > whether it is possible and would it make sense to adapt the SMA design to a > 16-antenna, dual-pol, 4096-channel system, or whether it is better (or > necessary) to leave the ROACH-2 designs behind and move to one of the newer > platforms? If the latter, what digitizer bandwidths are available, and > which board (SNAP2, Scarab, others?) would be most appropriate to a new > project of this scope? > > Thanks, > Dale > > -- > You received this message because you are subscribed to the Google Groups " > casper@lists.berkeley.edu" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to casper+unsubscr...@lists.berkeley.edu. > To post to this group, send email to casper@lists.berkeley.edu. > -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To post to this group, send email to casper@lists.berkeley.edu.
Re: [casper] Application of SMA correlator design to a larger array
Hi Dale The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are all using) would scale simply to your needs. You could easily build it out of ROACH2s or SKARABs at your scales. It would be a bit more expensive than the SWARM-type solution, in that it'd need additional hardware because it's not as resource-efficient as the SWARM design (you're going to struggle to beat SWARM with any other design!). A ROACH2 packetised solution would allow a number of different ADC options, including the the same one that SWARM uses. SKA-SA is moving in a slightly different direction with ADCs for our current and future systems. We have moved the ADC off the digital processing boards in the interests of reducing noise, and increasing overall system performance (constrain analogue bits to the antennas, with digital backhaul and processing to improve linearity and dynamic range, for example). We are not alone in our thinking here, but many in the CASPER community have not caught on to the benefits of this yet. And, unfortunately, it is a more expensive solution overall. At the moment, there are no universal/community off-board ADCs. Everyone's got really expensive, custom-designed ones (a possible gap in the CASPER lineup, here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we haven't really designed any ADCs to mate directly with them. At the moment, there is actually one ADC that plugs directly into a SKARAB, but I'm not sure it'd be appropriate for your needs, and it has no yellow block yet (though Peralex promised to deliver one at the last CASPER workshop if a customer wanted it). Another consideration is that the newer platforms (SKARAB, SNAP etc) are using the new JASPER toolflow, which is under active development and has support from the bigger CASPER developers. The older CASPER flow is still supported on ROACH2s, but new features and bug-fixes are not being back-ported and development there has stagnated. Since Xilinx will not support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be supported by all the new tools. I don't know of anyone using JASPER with ISE (though, it was possible at one time). If you wanted to build your correlator out of ROACH2s or SKARABs (they have similar processing capacities), a quick back-of-the-envelope, worst-case calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, 2GHz BW correlator (I can almost hear the infomercial already: "act now, and we'll throw in a free beamformer"; you'd fit a beam or two in the spare capacity within the X-engine boards, FWIW), a packetised design would need something like: 32x F-engine boards (though I'd say there's a good chance you could squeeze it into 16 boards). 32x X-engine boards (30 if you don't want to process the band edges) 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port switch with some loopback trickery, which would be much cheaper). It looks like you will be BRAM limited in both cases (otherwise you could halve the board counts). You could also opt for some BRAM-saving tradeoffs to ease fitment of two polarisations onto an F-engine. For example you could drop down to a 4-tap PFB, or reduce the delay-correction resolution (MeerKAT's specs, upon which I based the numbers above, are overkill for most applications). If you're using a single network switch, you might also be able to reduce packet buffer requirements, which currently use BRAM, too. The beauty of this design is in its flexibility. You can access the raw, intermediate data streams on the switch, swap out portions of the design for computers/GPUs (eg LEDA and HERA), add more antennas incrementally, increase or decreased processed bandwidth, change spectral resolution etc. all with a quick parameter change and a recompile. Jason Manley Functional Manager: DSP SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 07 Nov 2017, at 20:02, Jonathan Weintroub wrote: > Hi Dale, > > I’ll offer a few bullets on SWARM, the new SMA system. > > 1. SWARM is all open source and shared via CASPER and you are welcome to use > it as is, or develop it further to adapt it to a new application, indeed it > would be very pleasing to see the design used in some other instrument. > > 2. There is a paper which is worth reading to understand what SWARM is and > what it does. Take a careful look if you are contemplating using the design. > http://www.worldscientific.com/doi/pdf/10.1142/S2251171716410063 > You can get insight the paper without looking at the gory details of source > codes, both bitcodes and associated software, but if you want to dig even > deeper, sources are all shared here: > http://www.github.com/sma-wideband. > > 3. You are correct SWARM processes 2 GHz blocks of *usable” bandwidth. The > Nyquist band is somewhat wider, 2.288 GHz. That Nyquist band is divided > into 16,384 channels (not 1024), so in fact it exceeds (rather than falls > sho
Re: [casper] Application of SMA correlator design to a larger array
Hi Jason, Thanks for the quick estimate. If we are targeting 10 antennas, can't the correlator be made to work with only 10 F engines? The packets of the missing F engines would just not arrive at the X-engine, and hence the associated baselines would be zeroed or just ignored. We might also design the X-engine not to send blank data, and hence reduce the size of the switch needed. This might even allow us to halve the number of X-engine boards, but I am not sure about that. A 4-tap PFB is fine, and we will do fine delay correction off the boards (phase correction in software), so only coarse delay steps are needed. This is what we do for EOVSA, very successfully. We will probably include spectral kurtosis calculation, though, which means accumulating power-squared. Still, I think we have a shot at 1 F-engine board per antenna. Regards, Dale On Wed, Nov 8, 2017 at 2:24 AM, Jason Manley wrote: > Hi Dale > > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are > all using) would scale simply to your needs. You could easily build it out > of ROACH2s or SKARABs at your scales. It would be a bit more expensive than > the SWARM-type solution, in that it'd need additional hardware because it's > not as resource-efficient as the SWARM design (you're going to struggle to > beat SWARM with any other design!). A ROACH2 packetised solution would > allow a number of different ADC options, including the the same one that > SWARM uses. > > SKA-SA is moving in a slightly different direction with ADCs for our > current and future systems. We have moved the ADC off the digital > processing boards in the interests of reducing noise, and increasing > overall system performance (constrain analogue bits to the antennas, with > digital backhaul and processing to improve linearity and dynamic range, for > example). We are not alone in our thinking here, but many in the CASPER > community have not caught on to the benefits of this yet. And, > unfortunately, it is a more expensive solution overall. At the moment, > there are no universal/community off-board ADCs. Everyone's got really > expensive, custom-designed ones (a possible gap in the CASPER lineup, > here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we > haven't really designed any ADCs to mate directly with them. At the moment, > there is actually one ADC that plugs directly into a SKARAB, but I'm not > sure it'd be appropriate for your needs, and it has no yellow block yet > (though Peralex promised to deliver one at the last CASPER workshop if a > customer wanted it). > > Another consideration is that the newer platforms (SKARAB, SNAP etc) are > using the new JASPER toolflow, which is under active development and has > support from the bigger CASPER developers. The older CASPER flow is still > supported on ROACH2s, but new features and bug-fixes are not being > back-ported and development there has stagnated. Since Xilinx will not > support Virtex 6 in Vivado, ROACH2 (and earlier boards) can never be > supported by all the new tools. I don't know of anyone using JASPER with > ISE (though, it was possible at one time). > > If you wanted to build your correlator out of ROACH2s or SKARABs (they > have similar processing capacities), a quick back-of-the-envelope, > worst-case calculation suggests that, for a 16 dual-pol, 4k channel 8-tap, > 2GHz BW correlator (I can almost hear the infomercial already: "act now, > and we'll throw in a free beamformer"; you'd fit a beam or two in the spare > capacity within the X-engine boards, FWIW), a packetised design would need > something like: > > 32x F-engine boards (though I'd say there's a good chance you could > squeeze it into 16 boards). > 32x X-engine boards (30 if you don't want to process the band edges) > 1x Arista 7250QX-64 or similar ~64 port 40G switch (or just a 32-port > switch with some loopback trickery, which would be much cheaper). > > It looks like you will be BRAM limited in both cases (otherwise you could > halve the board counts). You could also opt for some BRAM-saving tradeoffs > to ease fitment of two polarisations onto an F-engine. For example you > could drop down to a 4-tap PFB, or reduce the delay-correction resolution > (MeerKAT's specs, upon which I based the numbers above, are overkill for > most applications). If you're using a single network switch, you might also > be able to reduce packet buffer requirements, which currently use BRAM, too. > > The beauty of this design is in its flexibility. You can access the raw, > intermediate data streams on the switch, swap out portions of the design > for computers/GPUs (eg LEDA and HERA), add more antennas incrementally, > increase or decreased processed bandwidth, change spectral resolution etc. > all with a quick parameter change and a recompile. > > Jason Manley > Functional Manager: DSP > SKA-SA > > Cell: +27 82 662 7726 > Work: +27 21 506 7300 > > On 07 Nov 2017, at 20:02, Jonathan Weintroub > wrote: >
Re: [casper] Application of SMA correlator design to a larger array
I can think of three ways to operate non-2^N antennas... 1) "fake" F-engine inputs: This is what MeerKAT does right now when it has fewer than 2^N antennas. We just multicast copies of redundant the data into unused F-engine inputs. It doesn't save you any hardware and is wasteful, because you're still calculating everything even though you're not using it. 2) omit unused X-engine inputs: Saves you the unnecessary F-engine boards over option 1. MeerKAT operates like this during a failure (eg an F-engine board fails during an observation). The system will continue to run with missing boards; you'll just lose the relevant baselines or frequency channels. Right now, if you try'n run the system with missing F-engines, a number of error counters will tick over to indicate faults. You would probably have to fiddle a bit with the registers and/or monitoring&control software to tweak it for your needs and avoid reporting these "false" failures if you plan to operate it like this nominally. 3) By far the neatest solution (the one the engineer in me begs for!) would be to edit the design to natively support non-2^N antennas. There are a few things that would need to be changed where we've taken "shortcuts" by just slicing out binary bits to figure out address mappings (eg IP address generation, buffer address lookups etc) which you'd have to calculate properly for non-2^N powers. But it's definitely do-able if you're up for a bit of development work. In fact, the CASPER X-engine core itself is not bound by 2^N powers. You can already ask it to calculate baselines for any number of antennas. The reorder block that follows it, though, is locked to 2^N antennas, as are packet buffers. The F-engine's output packetiser (which distributes data to the X-engines) also employs a bit-slicing mechanism and would need to be modified. These are actually not big changes for someone familiar with the design. SKA-SA may tackle #3 anyway, if we end up building a digital backend for SKA-1 (it'll have 190-odd antennas). Jason On 08 Nov 2017, at 14:12, Gary, Dale E. wrote: > Hi Jason, > > Thanks for the quick estimate. If we are targeting 10 antennas, can't the > correlator be made to work with only 10 F engines? The packets of the > missing F engines would just not arrive at the X-engine, and hence the > associated baselines would be zeroed or just ignored. We might also design > the X-engine not to send blank data, and hence reduce the size of the switch > needed. This might even allow us to halve the number of X-engine boards, but > I am not sure about that. > > A 4-tap PFB is fine, and we will do fine delay correction off the boards > (phase correction in software), so only coarse delay steps are needed. This > is what we do for EOVSA, very successfully. We will probably include > spectral kurtosis calculation, though, which means accumulating > power-squared. Still, I think we have a shot at 1 F-engine board per > antenna. > > Regards, > Dale > > On Wed, Nov 8, 2017 at 2:24 AM, Jason Manley wrote: > Hi Dale > > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are all > using) would scale simply to your needs. You could easily build it out of > ROACH2s or SKARABs at your scales. It would be a bit more expensive than the > SWARM-type solution, in that it'd need additional hardware because it's not > as resource-efficient as the SWARM design (you're going to struggle to beat > SWARM with any other design!). A ROACH2 packetised solution would allow a > number of different ADC options, including the the same one that SWARM uses. > > SKA-SA is moving in a slightly different direction with ADCs for our current > and future systems. We have moved the ADC off the digital processing boards > in the interests of reducing noise, and increasing overall system performance > (constrain analogue bits to the antennas, with digital backhaul and > processing to improve linearity and dynamic range, for example). We are not > alone in our thinking here, but many in the CASPER community have not caught > on to the benefits of this yet. And, unfortunately, it is a more expensive > solution overall. At the moment, there are no universal/community off-board > ADCs. Everyone's got really expensive, custom-designed ones (a possible gap > in the CASPER lineup, here?!). SKARAB (and the upcoming SKARAB-2) fit into > this model, and so we haven't really designed any ADCs to mate directly with > them. At the moment, there is actually one ADC that plugs directly into a > SKARAB, but I'm not sure it'd be appropriate for your needs, and it has no > yellow block yet (though Peralex promised to deliver one at the last CASPER > workshop if a customer wanted it). > > Another consideration is that the newer platforms (SKARAB, SNAP etc) are > using the new JASPER toolflow, which is under active development and has > support from the bigger CASPER developers. The old
Re: [casper] Application of SMA correlator design to a larger array
Hi Clifford and All, Thanks for your responses. After looking into our design further, it looks like we are moving toward a somewhat narrower bandwidth, which makes it compatible with the Skarab ADC. Since this is an entirely new platform for me, I'll need some help visualizing the architecture needed for a 10 antenna (or 16, if 2^n is required) dual-pol system. I assume that the paradigm of a packet-switched correlator has not changed, so we are talking about a set of F-engines feeding a set of X-engines. With 3, 4-input ADCs, we could get 6 dual-pol antennas into one Skarab, but are the FPGA resources able to handle so many F-engines? Can someone estimate for me what the practical limit (number of inputs/Skarab) would be, for a generic F-engine design? For our ROACH-2-based EOVSA correlator, we actually have both F- and X-engine functions on each ROACH-2, but what would be the preferred architecture in the Skarab case? To update the design concept from what I initially posted, we now want to digitize roughly 1 - 1.2 GHz (depends on how broadband we can make the horn feed) from 10 dual-pol antennas, channelizing to 4096 or more channels. We'll need ~10 ms dump times for the science case, which for baselines for 10 antennas comes to about 360 MB/s, if I count correctly, but if we have to dump baselines for all 16 inputs it becomes about 1 GB/s. This is no problem for a 40 Gbe system, I guess. Could anyone venture a ballpark guess as to the cost of a Skarab-based system to do this? Many Thanks, Dale On Thu, Nov 9, 2017 at 5:21 AM, Clifford van Dyk wrote: > Hi Dale > > In case you are interested in considering the Skarab platform, I have > attached the latest Skarab product glossies to this email (there is also > a lot if info on the Casper Skarab web pages). Currently, only one > 4-channel, 3 GSPS 14-bit ADC mezzanine exists, that can (practically) be > populated on any of the Skarab's 4 mezzanine sites. Since one site is > dedicated to 4 x 40 Gb Ethernet, this has a practical limit of 12 ADC > channels per Skarab. While the ADC is not capable of delivering a 2 GHz > instantaneous bandwidth you are after (Nyquist is limited to 1.5 GHz), > it has a number of other attractive features, like multiple built-in > DDCs behind each ADC channel. This may offload some of the high speed > trickery/resource that would otherwise go into the FPGA, opening up the > resource for other use. > > Alternatively, in keeping with the direction that SKA-SA is heading > w.r.t. external ADCs (Jason's comments below), the Skarab 4 x 40 GbE > Mezzanine can actually run any high speed SERDES protocol between the > QSFP+ connectors and the FPGA (there are no Ethernet-specific PHYs in > between - PHY is fully implemented in the FPGA). So while SKA-SA prefers > 40 GbE interfacing everywhere, it may be possible to use other SERDES > protocols to attach directly to external ADCs (e.g. JESD, ESIStream) > over the QSFP+ connectors - you can essentially see the mezzanine as > capable of carrying up to 160 Gbps over 16 configurable SERDES lanes. > > Let me know if you have any specific questions around the platform. > > Kind regards, > Clifford > > On 11/8/2017 9:24 AM, Jason Manley wrote: > > Hi Dale > > > > The CASPER packetised correlator (that PAPER/HERA/KAT-7/MeerKAT etc are > all using) would scale simply to your needs. You could easily build it out > of ROACH2s or SKARABs at your scales. It would be a bit more expensive than > the SWARM-type solution, in that it'd need additional hardware because it's > not as resource-efficient as the SWARM design (you're going to struggle to > beat SWARM with any other design!). A ROACH2 packetised solution would > allow a number of different ADC options, including the the same one that > SWARM uses. > > > > SKA-SA is moving in a slightly different direction with ADCs for our > current and future systems. We have moved the ADC off the digital > processing boards in the interests of reducing noise, and increasing > overall system performance (constrain analogue bits to the antennas, with > digital backhaul and processing to improve linearity and dynamic range, for > example). We are not alone in our thinking here, but many in the CASPER > community have not caught on to the benefits of this yet. And, > unfortunately, it is a more expensive solution overall. At the moment, > there are no universal/community off-board ADCs. Everyone's got really > expensive, custom-designed ones (a possible gap in the CASPER lineup, > here?!). SKARAB (and the upcoming SKARAB-2) fit into this model, and so we > haven't really designed any ADCs to mate directly with them. At the moment, > there is actually one ADC that plugs directly into a SKARAB, but I'm not > sure it'd be appropriate for your needs, and it has no yellow block yet > (though Peralex promised to deliver one at the last CASPER workshop if a > customer wanted it). > > > > Another consideration is that the newer platforms (SKARAB, SNAP etc) are