The issue of callbacks was discussed last year when we were first looking at this. The issue is needing to be able to initialize the persistent data at pool creation time. There are really only two viable approaches to this: Some sort of callback function, or else the application needs to allocate each entry in the pool and do the initialization. The problem of course, is that the latter is very awkward. It should be noted that DPDK uses the callback approach.
Here's what we said about this last year. We should discuss what we want to revisit of this and what we want to change, but it seems clear that this feature is needed for best performance. User Meta Data and Persistence In addition to system meta data associated with each buffer in a pool, ODP enables the application to reserve a number of bytes of user meta data for its own use. While ODP implementations MUST support user meta data, this data MAY or MAY NOT be persistent. Persistent means that it has a fixed implementation-assigned address that remains addressable to the application and retains its contents across allocates and frees of the buffer it is associated with. In this context persistence means that whatever was in the user meta data at the time the buffer was freed is still there when it is subsequently reallocated. By contrast, user meta data that is not persistent cannot be relied upon to survive a buffer free operation. In this case it is an application responsibility to ensure that if the user meta data itself contains pointers to other storage areas that they are appropriately freed or reference-counted prior to freeing the buffer and that the user meta data is re-initialized whenever a buffer is allocated from its containing buffer pool. The ODP_BUFFER_OPTS_UDATA_PERSIST option is used to request that user meta data be persistent. Note that implementations MAY provide persistent user meta data by default if they choose. This option is used by the application to indicate that it requires persistence. User meta data is meta data that the application will use in conjunction with the buffer, and is expected to be relatively small, though ODP does not specify a minimum or maximum size of user meta data that can be requested by an application. It is understood, however, that there MAY be performance penalties for large user meta data size requests, and implementations SHOULD document these so that applications may make appropriate trade-offs in deciding how much user meta data they can reasonably expect to use with a given ODP implementation. For example, an ODP implementation may be able to store up to X bytes of user meta data in a very efficient manner. If an application request a larger amount of user meta data this will result in the ODP implementation having to allocate a block of storage of the requested size and store a pointer to this block in an efficient manner. The result would be extra storage references to access the “extended” area and this MAY have a performance impact that the application SHOULD take into consideration in its design. When user meta data is requested, ODP will reserve the requested number of bytes of user meta data for each buffer in the pool that makes use of user meta data. From then the management of user meta data contents is an application responsibility. Applications that employ user meta data typically need to initialize it prior to use. There are two ways to do this: as part of buffer pool creation or as part of buffer processing following buffer allocation from the pool. User meta data initialization is accomplished by allowing the application to specify a user-supplied buffer initialization routine via the buf_init member of the odp_buffer_pool_init_t structure passed to to odp_buffer_pool_create() that has the type signature: typedef void (odp_buf_init_t)(odp_buffer_t buf, void *buf_init_arg); When this routine is called depends on whether the user meta data is persistent or not. for persistent user meta data, this routine is called for each buffer in the pool at the time the pool is created. At entry to buf_init(), the buffer has been reserved and any user meta data associated with in it has been cleared to zeros. This intent of this routine is to initialize buffer user meta data to other values as desired. If no buf_init routine is provided (i.e., the field is specified as NULL), user meta data will simply be cleared to zeros as part of default buffer pool initialization. If the user meta data is not persistent, then the supplied buf_init() routine is called as part of the processing of the odp_buffer_alloc() routine to initialize the user meta data for the buffer at the time it is allocated from the pool. For non-persistent user meta data, if no buf_init() routine is supplied then no initialization of the user meta data is performed at allocation time and the initialization and management of this data is entirely an application responsibility. Note that for predefined buffer pools, since these are created by the implementation during ODP initialization, the odp_buffer_pool_init_t used to control their initialization is passed as an argument to odp_init_global(). User meta data management is an application responsibility with the following provision. If as part of an ODP API call an ODP implementation replaces a buffer with an equivalent output buffer as part of its processing, then any non-persistent user meta data associated with the input buffer MUST be copied to the new buffer as part of the operation of the API. If the user meta data is persistent, then any such copy operation required is the responsibility of the application. For example, if an application queues a buffer to a crypto engine that returns a different buffer from the one supplied to it, then as part of the processing of that call, the underlying ODP implementation MUST copy the input buffer user meta data to the output buffer as part of its processing if the user meta data is non-persistent. The reason for this provision is that the application always has addressability to persistent user meta data even if a buffer is freed and so can perform whatever copying it needs from the original buffer upon receiving the replacement buffer. Applications can use the odp_buffer_is_same() routine to determine if two odp_buffer_t objects refer to the same buffer. On Fri, Jul 3, 2015 at 3:05 AM, Savolainen, Petri (Nokia - FI/Espoo) < petri.savolai...@nokia.com> wrote: > Maybe it's better to add this into packet pool parameters. Usage of build > time config macros should be minimized in the future. > > /** User area size in bytes. Specify as 0 if no user > area is needed. */ > uint32_t uarea_size; > > /** Static user area data. > * 0: User area data can be overwritten while in pool (between > free and alloc calls). > * 1: User area data is static. System will never overwrite the > area. > * Entire area is initialized to zero in pool creation. > * Default is 0. > */ > int uarea_static; > > > Every platform should be able to provided static area, but the performance > may degrade due to extra indirection etc. Zero initialization would help > user to tell if the area has valid data or not. I'd like to avoid callbacks > from pool create to keep it simple. > > -Petri > > > > -----Original Message----- > > From: ext Zoltan Kiss [mailto:zoltan.k...@linaro.org] > > Sent: Thursday, July 02, 2015 7:43 PM > > To: lng-odp@lists.linaro.org > > Cc: lng-odp-d...@lists.linaro.org; lng-odp-...@lists.linaro.org; > > Savolainen, Petri (Nokia - FI/Espoo) > > Subject: [API-NEXT PATCH RFC] api: packet: add optional persistent > > metadata > > > > This requirement came from ODP-OVS: it handles buffers through 'struct > > ofpbuf' > > (in upstream it became 'struct dp_packet'), and they can be non-ODP > > buffers as > > well. Therefore after receiving a packet currently we have to reset three > > fields in it: 'source' (to ODP), 'allocated' (segment size, OVS doesn't > > support > > multiple segments in this struct) and 'odp_pkt' (points to the containing > > odp_packet_t) > > The native DPDK implementation takes advantage of the fact that the > buffer > > and > > user metadata area, where this struct is stored is allocated during pool > > creation time, therefore it initializes it to static values at startup. > > When > > I've tried this behaviour with ODP-DPDK, the performance increased from > > ~12.7 > > to ~12.9 Mpp. That's only ~%1.6 increase, but in other words it's ~%17 of > > the > > gap between the current and OVS-DPDK performance (13.9 Mpp) > > I would like a propose an optional feature where the implementation can > > signal > > if it guarantees this behaviour. I think most implementations would > > preallocate these areas to achieve good performance, but e.g. linux- > > generic > > does not AFAIK. > > > > Signed-off-by: Zoltan Kiss <zoltan.k...@linaro.org> > > --- > > include/odp/api/config.h | 10 ++++++++++ > > include/odp/api/packet.h | 4 +++- > > 2 files changed, 13 insertions(+), 1 deletion(-) > > > > diff --git a/include/odp/api/config.h b/include/odp/api/config.h > > index 91ea34e..d48030e 100644 > > --- a/include/odp/api/config.h > > +++ b/include/odp/api/config.h > > @@ -121,6 +121,16 @@ extern "C" { > > */ > > #define ODP_CONFIG_PACKET_BUF_LEN_MAX (ODP_CONFIG_PACKET_SEG_LEN_MIN*6) > > > > +/** > > + * Guarantee that user metadata is allocated during pool creation > > + * > > + * If this parameter is 1, applications can avoid reseting persistent > > parts of > > + * the area returned by odp_packet_user_area() by setting them once, > > after pool > > + * creation. It will keep its content between odp_packet_free() and > > + * odp_packet_alloc(). > > + */ > > +#define ODP_CONFIG_PACKET_PERSISTENT_MD 0 > > + > > /** Maximum number of shared memory blocks. > > * > > * This the the number of separate SHM areas that can be reserved > > concurrently > > diff --git a/include/odp/api/packet.h b/include/odp/api/packet.h > > index 91a124a..d7aeb95 100644 > > --- a/include/odp/api/packet.h > > +++ b/include/odp/api/packet.h > > @@ -448,7 +448,9 @@ void odp_packet_user_ptr_set(odp_packet_t pkt, const > > void *ctx); > > * User area address > > * > > * Each packet has an area for user data. Size of the area is fixed and > > defined > > - * in packet pool parameters. > > + * in packet pool parameters. If ODP_CONFIG_PACKET_PERSISTENT_MD is 1, > > this > > + * area is allocated during pool creation, and keep its content between > > + * odp_packet_free() and odp_packet_alloc(). > > * > > * @param pkt Packet handle > > * > > -- > > 1.9.1 > > _______________________________________________ > lng-odp mailing list > lng-odp@lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lng-odp >
_______________________________________________ lng-odp mailing list lng-odp@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lng-odp
