This thread is to discuss ideas and proposed solutions to two issues
that have been raised by Sachin relating to VPP needs, as well as
Honnappa relating to the scalable scheduler.
Background
=========
ODP handles are abstract types that implementations may define to be
of arbitrary bit width in size. However, for a number of reasons
(e.g., provision of strong typing support, ABI compatibility,
efficiency of internal manipulation, etc.) these are typically
represented as 64-bit quantities. Some applications that store handles
in their own structures wish to minimize the cache footprint consumed
by these structures and so would like an option to store handles in a
more compact format that uses a smaller number of bits. To date
32-bits seems sufficient for application need, however in theory 16 or
even 8 bits might be desirable in some circumstances.
We already have an example of 8-bit handles in the odp_packet_seg_t
type, where odp-linux uses an 8-bit representation of this type as a
segment index when ODP is configured with --enable-abi-compat=no while
using a 64-bit size when configured with --enable-abi-compat=yes.
Considerations
============
In choosing the bit width to use in representing handles there are two
main considerations that implementations must take into account.
First, to achieve strong typing in C, handles need to be of pointer
width. For development this is a very valuable feature, which is why
implementations are encouraged to provide strong typing for ODP
abstract types.
Second, for ABI compatibility it is required that all implementations
use the same width for types that are to be ABI compatible across
different implementations. Implementations may interpret the bits of a
handle very differently, but all must agree that handles are of the
same bit width if they wish to be binary compatible with each other.
Stated Needs
===========
VPP currently packages its metadata into a vlib_mbuf struct that is
used pervasively to reference packets that are being processed by VPP
nodes. The address of this struct is desired to be held in compressed
(32-bit) format. Today the vlib_mbuf is implemented as a user area
associated with an odp_packet_t. As such the odp_packet_user_area()
API returns a (64-bit) pointer.
What is desired is a compact representation of this address.
VPP on the transmit side also needs to obtain the odp_packet_t
associated with a vlib_mbuf.
For the scalable scheduler, the desire is for a compact representation
of an odp_event_t that can be stored in a space-efficient manner in
queues.
Proposed Solutions
===============
Outlined here are a couple of proposed solutions to these problems.
Please feel free to propose alternate solutions as well.
For the case of the compact user area pointers needed by VPP, the
suggestion has been made that ODP pools provide an API to return pool
bounds information so that VPP can convert the user area pointers to a
more compact index. However, this makes a number of assumptions about
the internals of ODP pools that may or may not be portable or
practical in all implementations.
Since the requirement is for a compact representation of the user area
address, a more direct
solution may be simply to provide a set of new APIs that address this
need directly:
uint32_t odp_packet_user_area_index(odp_packet_t pkt);
This API would return a 32-bit index of the user area associated with
an odp_packet_t. Note that since user areas are mapped one-to-one with
ODP packets, this can serve effectively as a packet index as well.
With this API, applications can obtain the user area address directly
or an indirectly in a compact form. The problem is converting the
index back into the user area address. An API of the form:
void *odp_packet_user_area_addr(uint32_t ndx);
Assumes that this is a reversible mapping, which probably isn't true.
However, adding the odp_packet_t as a second argument would be
pointless since if the application has the odp_packet_t it can use the
existing odp_packet_user_area() API directly. So the containing pool
would seem a necessary 2nd argument:
void *odp_packet_user_area_addr(uint32_t ndx, odp_pool_t pool);
These APIs seem awkward as well, so perhaps recasting these as a way
to get compact packet handles might be better:
uint32_t odp_packet_to_index(odp_pool_t pool, odp_packet_t pkt);
odp_packet_t odp_packet_from_index(odp_pool_t pool, uint32_t ndx);
An interesting aside is that given this general approach, an
additional API could be envisioned that would provide even more
compact packet indexes:
uint16_t odp_packet_to_index_16(odp_pool_t pool, odp_packet_t pkt);
Note that a single odp_packet_from_index() suffices since uint16_t
indexes will promote to a uint32_t argument without problem. The
odp_pool_capability() API may indicate whether this additional compact
forms is supported, and of course this would only be possible if the
pool's pkt.num is < 64K.
With these APIs, the compact vlib_mbuf requirement would seem to be
satisfied by the following routines:
uint32_t vlib_mbuf_index(odp_packet_t pkt)
{
return odp_packet_to_index(odp_packet_pool(pkt), pkt);
}
void *vlib_mbuf_addr(odp_pool_t pool, uint32_t ndx)
{
return odp_packet_user_area(odp_packet_from_index(pool, ndx));
}
Conversely, the need to obtain the odp_packet_t from the vlib_mbuf
index would be simply:
odp_packet_t vlib_mbuf_to_pkt(odp_pool_t pool, uint32_t ndx)
{
return odp_packet_from_index(pool, ndx);
}
For the scalable scheduler, since this is internal to the ODP
implementation, there doesn't seem to be the need for any new external
APIs. Internal _odp_event_to_index() and _odp_event_from_index() APIs
could be modeled on this approach to achieve the same effect, however.
