Hey Dan and others, Thank you so much for your input. This is all very useful information.
To expand a little more, the RFSoC 4x2 is currently running a spectrometer. Our IF stage runs from 2-4 GHz, so the ADCs are sampling at 4096 MSPS. This data is read out using casperpga running on our flight computers(fanless embedded systems)<https://www.advantech.com/en/products/ark-2000_series_embedded_box_pcs/ark-2121l/mod_dd092808-0832-44bc-b38a-945eb7e016bd> and being stored directly onto SSDs. This part is pretty sorted out. The plan is to now add a UDP streaming design on the FPGA to store the complex voltage waveforms directly after digitizing and re-quantizing them to lower bits (as we won't need the entire 14 bits for correlating the data afterward). Depending on how much we end up re-quantizing our data, rates could be lower than 12 Gbps. Seeing that keeping the data rates below 10 Gbps makes things a lot simpler, I am planning to put a cap there. Mitch has been kind enough to share some of his test plans working with the 10 GbE port on the 4x2 and I am planning to purchase some of those components to test this plan out : 1. QSFP28 to SFP28 generic adapter (https://www.fs.com/products/178067.html) 2. 10G SFP+ Passive DAC cable (https://www.fs.com/products/30849.html?now_cid=2869) 3. A 10 GbE Mellanox NIC (https://a.co/d/6Ks4IAl) I was planning to interface this NIC with a more commercially available NAS like: https://www.qnap.com/en/product/ts-673a that seems to support PCIe Gen 3 slots. Maybe this is an overkill and something simple can be worked out here such as the minipcs listed on the spreadsheet. Being on a stratospheric balloon limits us from using fans so there will be a lot of thermal wizardry going on anyway. I hope that gives a little more context. Thanks again for all the suggestions so far, appreciate all the help. Regards, Mayukh ________________________________ From: 'Dan Werthimer' via casper@lists.berkeley.edu <casper@lists.berkeley.edu> Sent: Friday, September 27, 2024 6:29 PM To: casper@lists.berkeley.edu <casper@lists.berkeley.edu> Subject: Re: [casper] 100 GbE Hardware recommendation hi mayukh, another option to consider: some of the low power mini-PC computers in the spreadsheet have two 10Gbit/sec ethernet ports built in. (see ben's and my email below for the link to this spreadsheet, and look at the 10G ethernet column.) if you transmit your 12Gbit/sec FPGA data over a pair of 10Gbit ports, then you wouldn't need a NIC, and your mini-PC wouldn't need any PCIe slots. best wishes, dan On Fri, Sep 27, 2024 at 1:32 PM Dan Werthimer <dans...@berkeley.edu<mailto:dans...@berkeley.edu>> wrote: hi mayukh, if might help people guide you if you can you say a bit more about your experiment. what instrument and signal processing is running on the FPGA ? (spectrometer? correlator ? beamformer ? ) what is the FPGA output 12Gbit/sec data stream look like (spectra? complex channelized voltage data? ) what does the processor do with the 12Gbit/sec data stream after it receives it? store it on hard disks? SSD? (for how long?) compress the data and then store it ? compress the data and then transmit it to a ground station ? send uncompressed 12Gbit/sec to the ground via a very high speed data link ? can you compress the data on the FPGA ? instead of sending the data over 100/25/10 Gbit link to an external computer, can you instead process the data using the ARM core processors on the RFSOC ? there are six ARM cores on the RFSOC's FPGA. for info on these ARM processors, see: https://www.amd.com/en/corporate/university-program/aup-boards/rfsoc4x2.html if you need to send the data from the RFSOC board to an external computer over 100G, 25G, or a pair of 10G ethernet links, berkeley casper postdoc ben godfrey sent me a spreadsheet that lists a lot of low power mini-PC computers. (please see email from ben below). one of the column lists the number and type of PCIe slots, another column lists the power consumption of the board (you need to add the NIC power to this). another berkeley casper postdoc, wei liu, sent me an email about the PCIe slot on the Raspberry Pi 5. please see email from wei liu below ben's. best wishes, dan Ben Godfrey 1:07 PM (9 minutes ago) [https://mail.google.com/mail/u/1/images/cleardot.gif]Reply [https://mail.google.com/mail/u/1/images/cleardot.gif] to Wei, me [https://mail.google.com/mail/u/1/images/cleardot.gif] Not sure how useful this is, but here's a link to a Google Drive with a lot of mini PC's on it. Link: https://docs.google.com/spreadsheets/d/1SWqLJ6tGmYHzqGaa4RZs54iw7C1uLcTU_rLTRHTOzaA/edit?gid=239063037#gid=239063037 - Ben On Fri, Sep 27, 2024 at 1:02 PM Wei Liu <liuwei_berke...@berkeley.edu<mailto:liuwei_berke...@berkeley.edu>> wrote: Hi Dan, Ben, Here is an interesting blog about pcie on rpi5: https://www.jeffgeerling.com/blog/2023/forcing-pci-express-gen-30-speeds-on-pi-5 It looks like the pcie gen can be forced from 2.0 to 3.0. Someone tested 10G NIC on it, and got 6Gbps bandwidth. Best, Wei On Mon, Sep 23, 2024 at 11:33 AM Mayukh Bagchi <mayukh.bag...@queensu.ca<mailto:mayukh.bag...@queensu.ca>> wrote: Hey Dan, Thanks for getting back. My first concern is being able to find a compatible hardware/computer to plug the NIC into. Are these systems commercially available? Or are most of them custom-built? I could budget around 100 W for such a computer. I don't think the power draw of the NICs themselves is concerning for us, we could manage that. Also, I am not sure if the 100 GbE yellow block would be compatible with a dual 10 GbE NIC or a 40 GbE NIC, if it does then that opens up quite a few options. Regards, Mayukh ________________________________ From: 'Dan Werthimer' via casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu> <casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu>> Sent: Monday, September 23, 2024 1:48 PM To: casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu> <casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu>> Subject: Re: [casper] 100 GbE Hardware recommendation Hi mayukh, You might consider two 10 gbe links and a dual 10 gbe nic. Or a 40 gbe nic. Are you concerned with the power consumption of the 100 gbe nic, or the consumption of the computer the nic is plugged into? How much computing power and storage do you need ? Best wishes, Dan On Mon, Sep 23, 2024, 1:18 PM Mayukh Bagchi <mayukh.bag...@queensu.ca<mailto:mayukh.bag...@queensu.ca>> wrote: Hello Casperites, I hope you all have been doing well. I have been developing an RFSoC-based backend for our Balloon-borne VLBI Experiment (BVEX) project, which will be a mobile K-band VLBI station, launching in August next year. I am using the RFSoC 4x2 to digitize and stream data for our IF stage. The CASPER community has been very helpful with setup and troubleshooting. So far, I've successfully implemented a spectrometer and a power meter. While the design for the 100 GbE data streaming is almost ready, I am having trouble selecting a compatible NAS/server-side system to directly communicate with the 100 GbE Mellanox NIC<https://www.fs.com/products/119648.html?attribute=67745&id=3746425>. The reason why it is getting tricky is the size, thermal regulation, and power consumption constraints, as this will be on a balloon platform with limited resources. Our data rates should be around ~10-12 Gbps, depending on how much we end up re-quantizing, so we may not use the full capabilities of the 100 GbE network. However, at this point, any suggestion on hardware will be useful in getting us pointed in the right direction. Feel free to share your advice on working with 100 GbE Mellanox NICs. If this seems challenging, I could probably lower our data rates and try the 10GbE yellow block Ethernet core. Thanks in advance! Regards, Mayukh __________ Mayukh Bagchi (He/Him), PhD Candidate Department of Physics, Engineering Physics, and Astronomy mayukhbagchi.com<http://mayukhbagchi.com/> | mayukh.bag...@queensu.ca<mailto:mayukh.bag...@queensu.ca> [Queen's University Logo]<https://www.queensu.ca/> -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu>" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu<mailto:casper+unsubscr...@lists.berkeley.edu>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT4PR01MB10632111BE97CE7C079D9E886916F2%40YT4PR01MB10632.CANPRD01.PROD.OUTLOOK.COM<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT4PR01MB10632111BE97CE7C079D9E886916F2%40YT4PR01MB10632.CANPRD01.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu>" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu<mailto:casper+unsubscr...@lists.berkeley.edu>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAA9jdzJmp1E%3DZknwdB4V18WypD8ak0TewhDoqe8WsTSk1i1-YA%40mail.gmail.com<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAA9jdzJmp1E%3DZknwdB4V18WypD8ak0TewhDoqe8WsTSk1i1-YA%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu<mailto:casper@lists.berkeley.edu>" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu<mailto:casper+unsubscr...@lists.berkeley.edu>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT4PR01MB1063206055042E9DCB07FFFD2916F2%40YT4PR01MB10632.CANPRD01.PROD.OUTLOOK.COM<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT4PR01MB1063206055042E9DCB07FFFD2916F2%40YT4PR01MB10632.CANPRD01.PROD.OUTLOOK.COM?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu<mailto:casper+unsubscr...@lists.berkeley.edu>. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAA9jdzKD%3DrkarJ_NdAae_QJNqpVdQny1EaKe-aiDq5XjUad2ZQ%40mail.gmail.com<https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAA9jdzKD%3DrkarJ_NdAae_QJNqpVdQny1EaKe-aiDq5XjUad2ZQ%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/YT4PR01MB10632FD959EAC56EBEB30C43D916B2%40YT4PR01MB10632.CANPRD01.PROD.OUTLOOK.COM.
![https://mail.google.com/mail/u/1/images/cleardot.gif]Reply](https://mail.google.com/mail/u/1/images/cleardot.gif]Reply){kind=link}
