Hi Nigel,
I'll take a different slant:
In general, between components, the two buffer parameters (# buffers,
buffer size) are determined by default, and can be overriden (either in
the API or in the XML).
Remember that the number of buffers can be different at both ends of the
connection (what we call "asymmetrical multibuffering).
The default number of buffers is 2 at each side.
The buffer size default is based on the largest message in the protocol
defined for the connected ports.
This can be overridden, especially in the case where the message size is
unbounded (sequencelength=0).
If there is no protocol and no override, the default buffer size is 2K
bytes.
The PCI-OCDP:
- is used for a connection that goes between a FPGA worker to something
off chip via PCI.
- is attached to a memory that is fixed size, currently, as Shep said,
using BRAM with a parameter for how much.
- fully buffers OpenCPI messages in its memory.
Thus the number of buffers * buffer-size must fit in that memory.
We have three fairly long-standing roadmap enhancements for this:
1. Allow the OCDP to use DRAM as well as BRAM (and have multiple OCDP's
share a DRAM pool).
2. For better performance on CPU->FPGA flows, have the OCDP issue
multiple outstanding read requests to PCI.
3. Have a cut-through/FIFO-based OCDP that doesn't fully buffer the
messages (WSI vs WMI).
For CPU->FPGA flows (not FPGA->CPU flows), #2 is probably the limiting
throughput factor.
For now, increasing the message size up to the BRAM limit is a good thing.
Each buffer requires buffer_size + 32 bytes of the BRAM.
#1 is for more elasticity
#3 is more a latency enhancement (a.k.a.cut-through).
Jim
On 8/27/12 7:03 AM, Shepard Siegel wrote:
Hi Nigel,
I'll take a quick swipe at this and let others fill in the blanks and
(possibly) correct me.
The OCDP HDL PCIe dataplane engines are a specific HDL implementation
of the more general OCPI RDMA capability.
The PCie OCDPs use FPGA BRAMs for the message buffers that decouple
(allow concurrency) between the WIP Worker Message Interface (WMI)
to/from fabric (PCIe) message ingress and egress.
There is a `define that currently sets the BRAM size for each OCDP to
64KB.
So enough for several 4KB or 8KB buffers; but not enough for 4 or 8 MB.
Remember that "message length" is a property of a message, conveyed in
Bytes in the metadata, and unrelated to buffer size.
Any messages that are larger than buffer size need slicing/assembly
logic (some of which is in the PCIe OCDP) to yield PCIe messages that
satisfy the PCIe specifications for max transfer size and max read
request.
We know that MB, GB, TB transfers are common (even if they are "outer
loop" on some transport protocols) and are working on improving them.
-Shep
On Mon, Aug 27, 2012 at 3:22 AM, Merritt, Nigel (Contractor)
<[email protected]
<mailto:[email protected]>> wrote:
*UNCLASSIFIED*
We've been trying to increase the size of our buffer from RCC to
HDL space, due to the current speeds across PCI being fairly slow
(they roughly follow the small transfer speeds in Shep's email).
When we increased the sequenceLength in our protocol xml from "4k"
to "4m" (type=short thus 8192000 bytes) and run we get the message:
"/Requested buffer count and size won't fit in the OCDP's memory/"
Having tried various sizes, we clearly need some guidance on how
to create high BW transfers.
1. In Shep's email (a line from it is below), what is the
relationship between bytes and framesize to Type and
SequenceLength from protocol XML.
"/Bytes 16000, Time delta = 1947, 8.217771 MBytes/seconds,
Framesize 16/"
2. When a protocol contains several operations, does OpenCPI
allocate just enough space for the largest one or does it add them
together. I ask because the line in HdlContainer.cxx that checks
for an error and throws the OCDP error mentioned above has
"myDesc.nBuffers * (...)" which/implies/that the number of buffers
is multiplied by the largest buffer size. We have several
operations in one protocol and this may be causing issues.
3. If this isn't already answered by 1), what needs to be set in
order to have large buffer sizes (e.g. 8Mb), including OpenCPI
memory settings (e.g. allocation of memory for the OCDP).
Cheers,
Nigel
*IMPORTANT*: This email remains the property of the Department of
Defence and is subject to the jurisdiction of section 70 of the
Crimes Act 1914. If you have received this email in error, you are
requested to contact the sender and delete the email.
_______________________________________________
opencpi_dev mailing list
[email protected] <mailto:[email protected]>
http://lists.opencpi.org/listinfo.cgi/opencpi_dev-opencpi.org
_______________________________________________
opencpi_dev mailing list
[email protected]
http://lists.opencpi.org/listinfo.cgi/opencpi_dev-opencpi.org
_______________________________________________
opencpi_dev mailing list
[email protected]
http://lists.opencpi.org/listinfo.cgi/opencpi_dev-opencpi.org
