Hi Nigel, I'll take a quick swipe at this and let others fill in the blanks and (possibly) correct me. The OCDP HDL PCIe dataplane engines are a specific HDL implementation of the more general OCPI RDMA capability. The PCie OCDPs use FPGA BRAMs for the message buffers that decouple (allow concurrency) between the WIP Worker Message Interface (WMI) to/from fabric (PCIe) message ingress and egress. There is a `define that currently sets the BRAM size for each OCDP to 64KB. So enough for several 4KB or 8KB buffers; but not enough for 4 or 8 MB. Remember that "message length" is a property of a message, conveyed in Bytes in the metadata, and unrelated to buffer size. Any messages that are larger than buffer size need slicing/assembly logic (some of which is in the PCIe OCDP) to yield PCIe messages that satisfy the PCIe specifications for max transfer size and max read request. We know that MB, GB, TB transfers are common (even if they are "outer loop" on some transport protocols) and are working on improving them. -Shep
On Mon, Aug 27, 2012 at 3:22 AM, Merritt, Nigel (Contractor) < [email protected]> wrote: > ** > > *UNCLASSIFIED* > > We've been trying to increase the size of our buffer from RCC to HDL > space, due to the current speeds across PCI being fairly slow (they roughly > follow the small transfer speeds in Shep's email). > > When we increased the sequenceLength in our protocol xml from "4k" to > "4m" (type=short thus 8192000 bytes) and run we get the message: > "*Requested buffer count and size won't fit in the OCDP's memory*" > > Having tried various sizes, we clearly need some guidance on how to create > high BW transfers. > > 1. In Shep's email (a line from it is below), what is the relationship > between bytes and framesize to Type and SequenceLength from protocol XML. > > "*Bytes 16000, Time delta = 1947, 8.217771 MBytes/seconds, > Framesize 16*" > > 2. When a protocol contains several operations, does OpenCPI allocate just > enough space for the largest one or does it add them together. I ask > because the line in HdlContainer.cxx that checks for an error and throws > the OCDP error mentioned above has "myDesc.nBuffers * (…)" which*** > implies* that the number of buffers is multiplied by the largest buffer > size. We have several operations in one protocol and this may be causing > issues. > > 3. If this isn't already answered by 1), what needs to be set in order to > have large buffer sizes (e.g. 8Mb), including OpenCPI memory settings (e.g. > allocation of memory for the OCDP). > > Cheers, > > Nigel > > *IMPORTANT*: This email remains the property of the Department of Defence > and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If > you have received this email in error, you are requested to contact the > sender and delete the email. > > _______________________________________________ > opencpi_dev mailing list > [email protected] > http://lists.opencpi.org/listinfo.cgi/opencpi_dev-opencpi.org > >
_______________________________________________ opencpi_dev mailing list [email protected] http://lists.opencpi.org/listinfo.cgi/opencpi_dev-opencpi.org
