I know the comments are copied from the source, but thy are just copied
(not referenced), right?
I mean, if it does not get fixed here, there will be 2 places to fix... If
you have the courage, maybe the source comment can be fixed in a earlyer
patch.
These comments are relatively minor details. Fix what you can: This text is
much better that the emptiness that was before.
On 3 March 2016 at 17:12, Bill Fischofer <bill.fischo...@linaro.org> wrote:
>
>
> On Thu, Mar 3, 2016 at 4:00 AM, Christophe Milard <
> christophe.mil...@linaro.org> wrote:
>
>> On 2016-03-02 04:35, Bill Fischofer wrote:
>> > Signed-off-by: Bill Fischofer <bill.fischo...@linaro.org>
>> > ---
>> > doc/users-guide/users-guide-pktio.adoc | 605
>> +++++++++++++++++++++++++++++++++
>> > doc/users-guide/users-guide.adoc | 2 +
>> > 2 files changed, 607 insertions(+)
>> > create mode 100644 doc/users-guide/users-guide-pktio.adoc
>> >
>> > diff --git a/doc/users-guide/users-guide-pktio.adoc
>> b/doc/users-guide/users-guide-pktio.adoc
>> > new file mode 100644
>> > index 0000000..29fb6a9
>> > --- /dev/null
>> > +++ b/doc/users-guide/users-guide-pktio.adoc
>> > @@ -0,0 +1,605 @@
>> > +== PktIO Processing
>> > +Before packets can be manipulated they typically need to be _received_
>> and
>> > +after they are manipulated they need to be _transmitted_. The ODP
>> abstraction
>> > +that captures these operations is the *Packet I/O (PktIO)*.
>> > +PktIOs are represented by handles of type *odp_pktio_t* and
>> > +represent a logical I/O interface that is mapped in an
>> implementation-defined
>> > +manner to an underlying integrated I/O adapter or NIC.
>> > +
>> > +PktIO objects are manipulated through various state transitions via
>> > ++odp_pktio_xxx()+ API calls as shown below:
>> > +
>> > +.ODP PktIO Finite State Machine
>> > +image::../images/pktio_fsm.svg[align="center"]
>> > +
>> > +PktIOs begin in the *Unallocated* state. From here a call
>> +odp_pktio_open()+
>> > +is used to create an *odp_pktio_t* handle that is used in all
>> subsequent calls
>> > +to manipulate the object. This call puts the PktIO into the
>> *Unconfigured*
>> > +state. To become operational, a PktIO must first be
>> > +*configured* for Input, Output, or both Input and Output via the
>> > ++odp_pktin_queue_config()+ and/or +odp_pktout_queue_config()+ APIs,
>> and then
>> > +*started* via the +odp_pktio_start()+ to make it *Ready*.
>> > +
>> > +Following the completion of I/O processing, the +odp_pktio_stop()+ API
>> returns
>> > +the PktIO to the *Configured* state. From here it may be
>> *Reconfigured* via
>> > +additional +odp_pktin_queue_config()+ and/or
>> +odp_pktout_queue_config()+ calls,
>> > +or *Closed* via the +odp_pktio_close()+ API to return the PktIO to the
>> > +*Unallocated* state.
>> > +
>> > +=== PktIO Allocation
>> > +PktIO objects begin life by being _opened_ via the call:
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Open a packet IO interface
>> > + *
>> > + * An ODP program can open a single packet IO interface per device,
>> attempts
>>
>> what is a device here? a (possibly physical) interface? if we have a NIC
>> board with 2
>> physical inerfaces, how many pktio can on open on it?
>> maybe: "An ODP program can open a single packet IO per interface"...?
>>
>
> You're not commenting on this document, but on the
> include/odp/api/spec/packet_io.h file that I'm just excerpting snippets
> from here. We can improve that commentary, but that would be a separate
> patch. I'd prefer to keep these two separate at this point and make any
> doxygen changes as a separate patch series.
>
yes I know. see my comment on the top.
>
>> > + * to open an already open device will fail, returning
>> ODP_PKTIO_INVALID with
>> > + * errno set. Use odp_pktio_lookup() to obtain a handle to an already
>> open
>>
>> "opened" instead of open?
>>
>
> Same comment as above.
>
>
>>
>> > + * device. Packet IO parameters provide interface level configuration
>> options.
>> > + *
>> > + * Packet input queue configuration must be setup with
>> > + * odp_pktin_queue_config() before odp_pktio_start() is called. When
>> packet
>> > + * input mode is ODP_PKTIN_MODE_DISABLED, odp_pktin_queue_config()
>> call is
>> > + * optional and will ignore all parameters.
>> > + *
>> > + * Packet output queue configuration must be setup with
>> > + * odp_pktout_queue_config() before odp_pktio_start() is called. When
>> packet
>> > + * output mode is ODP_PKTOUT_MODE_DISABLED or ODP_PKTOUT_MODE_TM,
>> > + * odp_pktout_queue_config() call is optional and will ignore all
>> parameters.
>> > + *
>>
>> Is is not clear at this stage of reading what are the packet input and
>> output modes.
>> I assume that they are parameters to the pktio_open function (in param)
>> as this
>> is the only thing that is done before the configuration, maybe that should
>> be clarified before. (this assumption gets confirmed later on in the doc)
>> Simply adding "Pktio in/output mode will be discussed in the next
>> section" before
>> this code frgament would make sense.
>>
>
> Same comment as above.
>
>
>>
>> > + * Packet receive and transmit on the interface is enabled with a call
>> to
>> > + * odp_pktio_start(). If not specified otherwise, any interface level
>> > + * configuration must not be changed when the interface is active
>> (between start
>> > + * and stop calls).
>> > + *
>> > + * In summary, a typical pktio interface setup sequence is ...
>> > + * * odp_pktio_open()
>> > + * * odp_pktin_queue_config()
>> > + * * odp_pktout_queue_config()
>> > + * * odp_pktio_start()
>> > + *
>> > + * ... and tear down sequence is:
>> > + * * odp_pktio_stop()
>> > + * * odp_pktio_close()
>> > + *
>> > + * @param name Packet IO device name
>> > + * @param pool Default pool from which to allocate storage for
>> packets
>> > + * received over this interface, must be of type
>> ODP_POOL_PACKET
>> > + * @param param Packet IO parameters
>> > + *
>> > + * @return Packet IO handle
>> > + * @retval ODP_PKTIO_INVALID on failure
>> > + *
>> > + * @note The device name "loop" is a reserved name for a loopback
>> device used
>> > + * for testing purposes.
>> > + *
>> > + * @note Packets arriving via this interface assigned to a CoS by the
>> > + * classifier are received into the pool associated with that CoS.
>> This
>> > + * will occur either because this pktio is assigned a default CoS
>> via
>> > + * the odp_pktio_default_cos_set() routine, or because a matching
>> PMR
>> > + * assigned the packet to a specific CoS. The default pool specified
>> > + * here is applicable only for those packets that are not assigned
>> to a
>> > + * more specific CoS.
>> > + *
>> > + * @see odp_pktio_start(), odp_pktio_stop(), odp_pktio_close()
>> > + */
>> > +odp_pktio_t odp_pktio_open(const char *name, odp_pool_t pool,
>> > + const odp_pktio_param_t *param);
>> > +-----
>> > ++odp_pktio_open()+ takes three arguments: a *name*, which is an
>> > +implementation-defined string that identifies the logical interface to
>> be
>> > +opened, a *pool* that identifies the ODP pool that storage for received
>> > +packets should be allocated from, and a *param* structure that
>> specifies
>> > +I/O options to be associated with this PktIO instance.
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Packet IO parameters
>> > + *
>> > + * In minimum, user must select input and output modes. Use 0 for
>> defaults.
>> > + * Initialize entire struct with zero to maintain API compatibility.
>> > + */
>> > +typedef struct odp_pktio_param_t {
>> > + /** Packet input mode
>> > + *
>> > + * The default value is ODP_PKTIN_MODE_DIRECT. */
>> > + odp_pktin_mode_t in_mode;
>> > + /** Packet output mode
>> > + *
>> > + * The default value is ODP_PKTOUT_MODE_DIRECT. */
>> > + odp_pktout_mode_t out_mode;
>> > +} odp_pktio_param_t;
>> > +-----
>> > +ODP defines *loop* as a reserved name to indicate that this PktIO
>> represents
>>
>> I think "loop" should be quoted as it is a string.
>>
>
> It's bolded for that purpose, but I have no objection to adding quotes. Of
> course there will always be those who see and try to type the quotes and
> then wonder why it didn't work. :)
>
>>
>> > +a loopback interface. Loopback interfaces are useful as a means of
>> recycling
>> > +packets back for reclassification after decryption or decapsulation,
>> as well as
>> > +for diagnostic or testing purposes. The *pool* specifies the default
>> pool to
>>
>> "Loopback interfaces are useful as a means of recycling packets back for
>> reclassification after decryption or decapsulation". I don't understand
>> what you
>> mean by that... should I? ;-)
>>
>
> These are two common reasons why you'd use a loopback interface in
> non-testing mode. Until you decrypt an IPSEC packet you really have no
> idea what's really in it. Similar considerations apply to tunneled packets
> that need to be decapsulated before the "real" packet is visible.
>
I understand that, but I cannot see how a loopback interface would help
here. You can explain at BKK !
>
>
>>
>> > +use for packet allocation if not overridden by the classifier due to a
>> > +specific or default Class-of-Service (CoS) match on the packet. The
>> *param*
>> > +struct, in turn, specifies the input and output *modes* of the PktIO.
>> > +
>> > +=== PktIO Input and Output Modes
>> > +PktIO objects support four different Input and Output modes, that may
>> be
>> > +specified independently at *open* time.
>> > +
>> > +.PktIO Input Modes
>> > +* +ODP_PKTIN_MODE_DIRECT+
>> > +* +ODP_PKTIN_MODE_QUEUE+
>> > +* +ODP_OKTIN_MODE_SCHED+
>> > +* +ODP_PKTIN_MODE_DISABLED+
>> > +
>> > +.PktIO Output Modes
>> > +* +ODP_PKTOUT_MODE_DIRECT+
>> > +* +ODP_PKTOUT_MODE_QUEUE+
>> > +* +ODP_PKTOUT_MODE_TM+
>> > +* +ODP_PKTOUT_MODE_DISABLED+
>> > +
>> > +The DISABLED modes indicate that either input or output is prohibited
>> on this
>> > +PktIO. Attempts to receive packets on a PktIO whose +in_mode+ is
>> DISABLED
>> > +return no packets while packets sent to a PktIO whose +out_mode+ is
>> DISABLED
>> > +are discarded.
>> > +
>> > +==== Direct I/O Modes
>> > +DIRECT I/O is the default mode for PktIO objects. It is designed to
>> support
>> > +poll-based packet processing, which is often found in legacy
>> applications
>> > +being ported to ODP, and can also be a preferred mode for some types of
>> > +packet processing. By supporting poll-based I/O processing, ODP
>> provides
>> > +maximum flexibility to the data plane application writer.
>> > +
>> > +===== Direct RX Processing
>> > +The processing of DIRECT input is shown below:
>> > +
>> > +.PktIO DIRECT Mode Receive Processing
>> > +image::../images/pktin_direct_recv.svg[align="center"]
>> > +
>>
>> From this picture, I understand that a each rx pktio queue is "associated"
>> to a RX thread at odp_pktin_queue_config time. however, it does not feel
>> like
>> that at all in the rest of the document: odp_pktin_recv() take a queue
>> handle
>> as parameter, hence allowing any ODP thread to read packets from any queue
>> at any time... (the "operation mode", -not really defined for RX- also
>> implies
>> that such an association does not exist, at least in unsafe mode)
>>
>
> That is correct. The intent is to allow applications to have a one-to-one
> map between threads and RX queues, but ODP does not constrain applications
> to follow such a design. odp_pktin_queue_config() simply says how many RX
> queues to create/associated with the PktIO. The discipline of how threads
> use these queues is up to the application. I can add another paragraph
> here if that isn't clear in context.
>
My understanding is that ODP does not impose any queue-thread relationship,
but the application will probably want to have this queue-thread
relationship. This relationship is actually visible at 2 places as I
understand it: When reading packet from the queue (the queue ID is passed)
and when selecting the safe/unsafe mode (choosing safe meaning that One
queue will be read by a single thread). I don't see where the
odp_pktin_queue_config() function is involved at all in the relationship
(appart for configuring N queues, but no relation is made there right?).
>> > +In DIRECT mode, received packets are stored in one or more special
>> PktIO queues
>> > +of type *odp_pktin_queue_t* and are retrieved by threads calling the
>> > ++odp_pktin_recv()+ API.
>> > +
>> > +Once opened, setting up a DIRECT mode PktIO is performed by the
>> > ++odp_pktin_queue_config()+ API.
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Configure packet input queues
>> > + *
>> > + * Setup a number of packet input queues and configure those. The
>> maximum number
>> > + * of queues is platform dependent and can be queried with
>> > + * odp_pktio_capability(). Use odp_pktin_queue_param_init() to
>> initialize
>> > + * parameters into their default values. Default values are also used
>> when
>> > + * 'param' pointer is NULL.
>> > + *
>> > + * Queue handles for input queues can be requested with
>> odp_pktin_queue() or
>> > + * odp_pktin_event_queue() after this call. All requested queues are
>> setup on
>> > + * success, no queues are setup on failure. Each call reconfigures
>> input queues
>> > + * and may invalidate all previous queue handles.
>> > + *
>> > + * @param pktio Packet IO handle
>> > + * @param param Packet input queue configuration parameters. Uses
>> defaults
>> > + * when NULL.
>> > + *
>> > + * @retval 0 on success
>> > + * @retval <0 on failure
>> > + *
>> > + * @see odp_pktio_capability(), odp_pktin_queue(),
>> odp_pktin_event_queue()
>> > + */
>> > +int odp_pktin_queue_config(odp_pktio_t pktio,
>> > + const odp_pktin_queue_param_t *param);
>> > +-----
>> > +The second argument to this call is the *odp_pktin_queue_param_t*
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Packet input queue parameters
>> > + */
>> > +typedef struct odp_pktin_queue_param_t {
>> > + /** Operation mode
>> > + *
>> > + * The default value is ODP_PKTIO_OP_MT. Application may enable
>> > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE
>> when
>> > + * applicable. */
>> > + odp_pktio_op_mode_t op_mode;
>>
>> Has the "operation mode" been defined anywhere?
>> not sure what it is. neither do I
>> know what ODP_PKTIO_OP_MT means at this stage of reading...
>> This is defined only when talking about TX, later on...
>> It should be defined here (and possibly reminded for TX)
>>
>
> Doxygen comments again--not part of this patch series
>
>
>>
>> > +
>> > + /** Enable flow hashing
>> > + * 0: Do not hash flows
>> > + * 1: Hash flows to input queues */
>> > + odp_bool_t hash_enable;
>> > +
>> > + /** Protocol field selection for hashing. Multiple protocols can
>> be
>> > + * selected. */
>> > + odp_pktin_hash_proto_t hash_proto;
>> > +
>> > + /** Number of input queues to be created. More than one input
>> queue
>> > + * require input hashing or classifier setup. Hash_proto is
>> ignored
>> > + * when hash_enable is zero or num_queues is one. This value
>> must be
>> > + * between 1 and interface capability. Queue type is defined by
>> the
>> > + * input mode. The default value is 1. */
>> > + unsigned num_queues;
>> > +
>> > + /** Queue parameters for creating input queues in
>> ODP_PKTIN_MODE_QUEUE
>> > + * or ODP_PKTIN_MODE_SCHED modes. Scheduler parameters are
>> considered
>> > + * only in ODP_PKTIN_MODE_SCHED mode. */
>> > + odp_queue_param_t queue_param;
>> > +
>> > +} odp_pktin_queue_param_t;
>> > +-----
>> > +Note that the *queue_param* field of this struct is ignored in DIRECT
>> mode.
>> > +The purpose of +odp_pktin_queue_config()+ is to specify the number of
>> PktIn
>> > +queues to be created and to set their attributes.
>> > +
>> > +===== Hash Processing
>> > +Another feature of DIRECT mode input is the provision of a *hash*
>> function used
>> > +to distribute incoming packets among the PktIO's PktIn queues. If the
>> > ++hash_enable+ field of the *odp_pktin_queue_param_t* is 1,
>> > +then the +hash_proto+ field is used to specify which field(s) of
>> incoming
>> > +packets should be used as input to an implementation-defined packet
>> > +distribution hash function.
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Packet input hash protocols
>> > + *
>> > + * The list of protocol header field combinations, which are included
>> into
>> > + * packet input hash calculation.
>> > + */
>> > +typedef union odp_pktin_hash_proto_t {
>> > + /** Protocol header fields for hashing */
>> > + struct {
>> > + /** IPv4 addresses and UDP port numbers */
>> > + uint32_t ipv4_udp : 1;
>> > + /** IPv4 addresses and TCP port numbers */
>> > + uint32_t ipv4_tcp : 1;
>> > + /** IPv4 addresses */
>> > + uint32_t ipv4 : 1;
>> > + /** IPv6 addresses and UDP port numbers */
>> > + uint32_t ipv6_udp : 1;
>> > + /** IPv6 addresses and TCP port numbers */
>> > + uint32_t ipv6_tcp : 1;
>> > + /** IPv6 addresses */
>> > + uint32_t ipv6 : 1;
>> > + } proto;
>> > +
>> > + /** All bits of the bit field structure */
>> > + uint32_t all_bits;
>> > +} odp_pktin_hash_proto_t;
>> > +-----
>> > +Note that the hash function used in PktIO poll mode operation is
>> intended to
>> > +provide simple packet distribution among multiple PktIn queues
>> associated with
>> > +the PktIO. It does not have the sophistication of the *ODP
>> Classifier*, however
>> > +it also does not incur the setup requirements of pattern matching
>> rules,
>> > +making it a simpler choice for less sophisticated applications. Note
>> that
>> > +ODP does not specify how the hash is to be performed. That is left to
>> each
>> > +implementation. The hash only specifies which input packet fields are
>> of
>> > +interest to the application and should be considered by the hash
>> function in
>> > +deciding how to distribute packets among PktIn queues. The only
>> expectation
>> > +is that packets that have the same hash values should all be mapped to
>> the
>> > +same PktIn queue.
>> > +
>> > +===== PktIn Queues
>> > +A *PktIn Queue* is a special type of queue that is used internally by
>> PktIOs
>> > +operating in DIRECT mode. Applications cannot perform enqueues to
>> these queues,
>> > +however they may obtain references to them via the +odp_pktin_queue()+
>> API
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Direct packet input queues
>> > + *
>> > + * Returns the number of input queues configured for the interface in
>> > + * ODP_PKTIN_MODE_DIRECT mode. Outputs up to 'num' queue handles when
>> the
>> > + * 'queues' array pointer is not NULL. If return value is larger than
>> 'num',
>> > + * there are more queues than the function was allowed to output. If
>> return
>> > + * value (N) is less than 'num', only queues[0 ... N-1] have been
>> written.
>> > + *
>> > + * Packets from these queues are received with odp_pktin_recv().
>> > + *
>> > + * @param pktio Packet IO handle
>> > + * @param[out] queues Points to an array of queue handles for output
>> > + * @param num Maximum number of queue handles to output
>> > + *
>> > + * @return Number of packet input queues
>> > + * @retval <0 on failure
>> > + */
>> > +int odp_pktin_queue(odp_pktio_t pktio, odp_pktin_queue_t queues[], int
>> num);
>> > +-----
>> > +Once configured, prior to receiving packets the PktIO must be placed
>> into the
>> > +*Ready* state via a call to +odp_pktio_start()
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Start packet receive and transmit
>> > + *
>> > + * Activate packet receive and transmit on a previously opened or
>> stopped
>> > + * interface. The interface can be stopped with a call to
>> odp_pktio_stop().
>> > + *
>> > + * @param pktio Packet IO handle
>> > + *
>> > + * @retval 0 on success
>> > + * @retval <0 on failure
>> > + *
>> > + * @see odp_pktio_open(), odp_pktio_stop()
>> > + */
>> > +int odp_pktio_start(odp_pktio_t pktio);
>> > +-----
>> > +Once started, the PktIn queue handles are used as arguments to
>> > ++odp_pktin_recv()+ to receive packets from the PktIO.
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Receive packets directly from an interface input queue
>> > + *
>> > + * Receives up to 'num' packets from the pktio interface input queue.
>> When
>> > + * input queue parameter 'op_mode' has been set to
>> ODP_PKTIO_OP_MT_UNSAFE,
>> > + * the operation is optimized for single thread operation per queue
>> and the same
>> > + * queue must not be accessed simultaneously from multiple threads.
>> > + *
>> > + * @param queue Packet input queue handle for receiving
>> packets
>> > + * @param[out] packets[] Packet handle array for output of received
>> packets
>> > + * @param num Maximum number of packets to receive
>> > + *
>> > + * @return Number of packets received
>> > + * @retval <0 on failure
>> > + *
>> > + * @see odp_pktin_queue()
>> > + */
>> > +int odp_pktin_recv(odp_pktin_queue_t queue, odp_packet_t packets[],
>> int num);
>>
>> This means that any thread can read from any queue at any time, right?
>> (does not match my understanding of figure 13)
>>
>
> That is correct. As noted above, ODP just provides the tools.
> Applications must use these tools intelligently.
>
>
>>
>> > +-----
>> > +
>> > +===== Direct TX Processing
>> > +A PktIO operating in DIRECT mode performs TX processing as shown here:
>> > +
>> > +.PktIO DIRECT Mode Transmit Processing
>> > +image::../images/pktout_direct_send.svg[align="center"]
>> > +
>> > +Direct TX processing operates similarly to Direct RX processing.
>> Following
>> > +open, the +odp_pktout_queue_config()+ API is used to create and
>> configure
>> > +one or more *PktOut queues* to be used to support packet transmission
>> by
>> > +this PktIO
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Configure packet output queues
>> > + *
>> > + * Setup a number of packet output queues and configure those. The
>> maximum
>> > + * number of queues is platform dependent and can be queried with
>> > + * odp_pktio_capability(). Use odp_pktout_queue_param_init() to
>> initialize
>> > + * parameters into their default values. Default values are also used
>> when
>> > + * 'param' pointer is NULL.
>> > + *
>> > + * Queue handles for output queues can be requested with
>> odp_pktout_queue() or
>> > + * odp_pktout_event_queue() after this call. All requested queues are
>> setup on
>> > + * success, no queues are setup on failure. Each call reconfigures
>> output queues
>> > + * and may invalidate all previous queue handles.
>> > + *
>> > + * @param pktio Packet IO handle
>> > + * @param param Packet output queue configuration parameters. Uses
>> defaults
>> > + * when NULL.
>> > + *
>> > + * @retval 0 on success
>> > + * @retval <0 on failure
>> > + *
>> > + * @see odp_pktio_capability(), odp_pktout_queue(),
>> odp_pktout_event_queue()
>> > + */
>> > +int odp_pktout_queue_config(odp_pktio_t pktio,
>> > + const odp_pktout_queue_param_t *param);
>> > +-----
>> > +As with +odp_pktin_queue_config()+, the configuration of PktOut queues
>> > +involves the use of a parameter struct:
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Packet output queue parameters
>> > + *
>> > + * These parameters are used in ODP_PKTOUT_MODE_DIRECT and
>> > + * ODP_PKTOUT_MODE_QUEUE modes.
>> > + */
>> > +typedef struct odp_pktout_queue_param_t {
>> > + /** Operation mode
>> > + *
>> > + * The default value is ODP_PKTIO_OP_MT. Application may enable
>> > + * performance optimization by defining ODP_PKTIO_OP_MT_UNSAFE
>> when
>> > + * applicable. */
>> > + odp_pktio_op_mode_t op_mode;
>> > +
>> > + /** Number of output queues to be created. The value must be
>> between
>> > + * 1 and interface capability. The default value is 1. */
>> > + unsigned num_queues;
>> > +
>> > +} odp_pktout_queue_param_t;
>> > +-----
>> > +As with direct input, direct output uses one or more special output
>> queues
>> > +of type *odp_pktout_queue_t* that area created and configured by this
>> call.
>>
>> typo: "area" / "are"
>>
>>
> Good catch. Thanks.
>
>
>> > +
>> > +As with PktIn queues, the handles for these created PktOut queues may
>> be
>> > +retrieved by the +odp_pktout_queue()+ API:
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Direct packet output queues
>> > + *
>> > + * Returns the number of output queues configured for the interface in
>> > + * ODP_PKTOUT_MODE_DIRECT mode. Outputs up to 'num' queue handles when
>> the
>> > + * 'queues' array pointer is not NULL. If return value is larger than
>> 'num',
>> > + * there are more queues than the function was allowed to output. If
>> return
>> > + * value (N) is less than 'num', only queues[0 ... N-1] have been
>> written.
>> > + *
>> > + * Packets are sent to these queues with odp_pktout_send().
>> > + *
>> > + * @param pktio Packet IO handle
>> > + * @param[out] queues Points to an array of queue handles for output
>> > + * @param num Maximum number of queue handles to output
>> > + *
>> > + * @return Number of packet output queues
>> > + * @retval <0 on failure
>> > + */
>> > +int odp_pktout_queue(odp_pktio_t pktio, odp_pktout_queue_t queues[],
>> int num);
>> > +-----
>> > +Once the PktIO has been configured for output and started via
>> > ++odp_pktio_start()+, packets may be transmitted to the PktIO by calling
>> > +_odp_pktout_send()+:
>>
>> "+" missing before _odp_pktout_send()+. (The + you see is from the patch!)
>>
>
> Thank you.
>
>
>>
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Send packets directly to an interface output queue
>> > + *
>> > + * Sends out a number of packets to the interface output queue. When
>> > + * output queue parameter 'op_mode' has been set to
>> ODP_PKTIO_OP_MT_UNSAFE,
>> > + * the operation is optimized for single thread operation per queue
>> and the same
>> > + * queue must not be accessed simultaneously from multiple threads.
>>
>> The above should be stated when discussing RX!
>>
>
> Noted. v3 will add additional detail to the narrative.
>
>
>>
>> > + *
>> > + * A successful call returns the actual number of packets sent. If
>> return value
>> > + * is less than 'num', the remaining packets at the end of packets[]
>> array
>> > + * are not consumed, and the caller has to take care of them.
>> > + *
>> > + * @param queue Packet output queue handle for sending packets
>> > + * @param packets[] Array of packets to send
>> > + * @param num Number of packets to send
>> > + *
>> > + * @return Number of packets sent
>> > + * @retval <0 on failure
>> > + */
>> > +int odp_pktout_send(odp_pktout_queue_t queue, odp_packet_t packets[],
>> int num);
>> > +-----
>> > +Note that the argument to this call specifies the PktOut queue to that
>> the
>>
>> is "to that" english? (was expecting "to which") here. If it is, ignore
>> this
>> comment!
>>
>
> "to" is an extraneous word here. Good catch.
>
>
>>
>> > +packet is to be added to rather than the PktIO itself. This permits
>> multiple
>> > +threads (presumably operating on different cores) a more efficient
>> means of
>> > +separating I/O processing destined for the same interface.
>> > +
>> > +==== Queued I/O Modes
>> > +To provide additional flexibility when operating in poll mode, PktIOs
>> may also
>> > +be opened in QUEUE Mode. The difference between DIRECT and QUEUE mode
>> is that
>> > +QUEUE mode uses standard ODP event queues to service packets.
>> > +
>> > +===== Queue RX Processing
>> > +The processing for QUEUE input processing is shown below:
>> > +
>> > +.PktIO QUEUE Mode Receive Processing
>> > +image::../images/pktin_queue_recv.svg[align="center"]
>>
>> same comment: this figure seems to imply that one can create an
>> association
>> between the queues and the threads at odp_pktin_qyeye_config time...
>>
>
> No, it's showing how the created queues can be used most efficiently.
> I'll expand the comments around this in v3.
>
>
>>
>> > +
>> > +In QUEUE mode, received packets are stored in one or more standard ODP
>> queues.
>> > +The difference is that these queues are not created directly by the
>> > +application. Instead, they are created in response to an
>> > ++odp_pktin_queue_config()+ call.
>> > +
>> > +As with DIRECT mode, the +odp_pktin_queue_param_t+ specified to this
>> call
>> > +indicates whether an input hash should be used and if so which
>> field(s) of
>> > +the packet should be considered as input to the has function.
>> > +
>> > +The main difference between DIRECT and QUEUE RX processing is that
>> because
>> > +the PktIO uses standard ODP event queues, other parts of the
>> application can
>> > +use +odp_queue_enq()+ API calls to enqueue packets to these queues for
>> > +"RX" processing in addition to those originating from the PktIO
>> interface
>> > +itself. To obtain the handles of these input queues, the
>> > ++odp_pktin_event_queue()+ API is used:
>> > +[source,c]
>> > +-----
>> > +/**
>> > + * Event queues for packet input
>> > + *
>> > + * Returns the number of input queues configured for the interface in
>> > + * ODP_PKTIN_MODE_QUEUE and ODP_PKTIN_MODE_SCHED modes. Outputs up to
>> 'num'
>> > + * queue handles when the 'queues' array pointer is not NULL. If
>> return value is
>> > + * larger than 'num', there are more queues than the function was
>> allowed to
>> > + * output. If return value (N) is less than 'num', only queues[0 ...
>> N-1] have
>> > + * been written.
>> > + *
>> > + * Packets (and other events) from these queues are received with
>> > + * odp_queue_deq(), odp_schedule(), etc calls.
>> > + *
>> > + * @param pktio Packet IO handle
>> > + * @param[out] queues Points to an array of queue handles for output
>> > + * @param num Maximum number of queue handles to output
>> > + *
>> > + * @return Number of packet input queues
>> > + * @retval <0 on failure
>> > + */
>> > +int odp_pktin_event_queue(odp_pktio_t pktio, odp_queue_t queues[], int
>> num);
>> > +-----
>> > +Similarly, threads receive packets from PktIOs operating in QUEUE mode
>> by
>> > +making standard +odp_queue_deq()+ calls to one of the event queues
>> associated
>> > +with the PktIO.
>> > +
>> > +===== Queue TX Processing
>> > +Transmit processing for PktIOs operating in QUEUE mode is shown below:
>> > +
>> > +.PktIO QUEUE Mode Transmit Processing
>> > +image::../images/pktout_queue_send.svg[align="center]
>> > +
>> > +For TX processing QUEUE mode behaves similar to DIRECT mode except that
>> > +output queues are regular ODP event queues that receive packets via
>> > ++odp_queue_enq()+ calls rather than special PktOut queues that use
>> > ++odp_pktout_send()+. Again, these queues are created via a call to
>> > ++odp_pktout_queue_config()+ following +odp_pktio_open()+.
>> > +
>> > +The main reason for selecting QUEUE mode for output is flexibility. If
>> an
>> > +application is designed to use a _pipeline model_ where packets flow
>> through
>> > +a series of processing stages via queues, then having the PktIO in
>> QUEUE
>> > +mode means that the application can always use the same enq APIs to
>> pass packets
>> > +from one stage to the next, including the final transmit output stage.
>> > +
>> > +==== Scheduled I/O Modes
>> > +The final PktIO mode supported integrates RX and TX processing with
>> the ODP
>> > +_event model_. For RX processing this involves the use of the
>> *Scheduler*
>> > +while for TX processing this involves the use of the *Traffic Manager*.
>> > +
>> > +Scheduled RX Processing is further divided based on whether or not the
>> > +Classifier is used.
>> > +
>> > +===== Scheduled RX Processing
>> > +When a PktIO is opened with +ODP_PKTIN_MODE_SCHED+, it indicates that
>> the
>> > +input queues created by a subsequent +odp_pktin_queue_config()+ call
>> are to
>> > +be used as input to the *ODP Scheduler*.
>> > +
>> > +.PktIO SCHED Mode Receive Processing
>> > +image::../images/pktin_sched_recv.svg[align="center']
>> > +
>> > +For basic use, SCHED mode simply associates the PktIO input event
>> queues
>> > +created by +odp_pktin_queue_config()+ with the scheduler. Hashing may
>> still be
>> > +employed to distribute input packets among multiple input queues.
>> However
>> > +instead of these being plain queues they are scheduled queues and have
>> > +associated scheduling attributes like priority, scheduler group, and
>> > +synchronization mode (parallel, atomic, ordered). SCHED mode thus
>> provides
>> > +both packet distribution (via the optional hash) as well as
>> scalability via
>> > +the ODP event model.
>> > +
>> > +In its fullest form, PktIOs operating in SCHED mode use the *ODP
>> Classifier*
>> > +to permit fine-grained flow separation on *Class of Service (CoS)*
>> boundaries.
>> > +
>> > +.PktIO SCHED Mode Receive Processing with Classification
>> > +image::../images/pktin_sched_cls.svg[align="center"]
>>
>> The diagram and the text confuses me a bit, here: my understanding was
>> that:
>> 1)the classifier associates a cos to each incoming packet using a set of
>> PMRs.
>> 2)Each cos maps to a pair {RXpool, RXqueue}. This Mapping is settled
>> during the
>> classifier setup. Packets with a given cos are stored on odp_packets
>> allocated
>> from the corresponding pool, and placed in the corresponding queue.
>>
>
> That is correct. What's unclear about that here? The point is that the
> queues shown here may either be those that the application creates via it's
> own odp_queue_create() calls or those created by odp_pktin_queue_config().
> If the latter, the application will use odp_pktin_queue() to retrieve their
> handle to be passed when it creates the CoS it's to be associated with.
> Remember, multiple CoSes can add packets to the same queue--this is not a
> 1-to-1 association here.
>
Maybe the figure wants to show too much. I don't really see what's going on
there. But maybe it is part of the classifier.
Anyway. see U at BKK!
/Christophe.
>
>
>>
>> > +
>> > +In this mode of operation, the hash function of
>> +odp_pktin_queue_config()+ is
>> > +not used. Instead, the event queues created by this call, as well as
>> any
>> > +additional event queues created via separate +odp_queue_create()+
>> calls are
>> > +associated with classes of service via +odp_cls_cos_create()+ calls.
>> > +
>> > +===== Scheduled TX Processing
>> > +Scheduled transmit processing is performed via the *ODP Traffic
>> Manager* and
>> > +is requested when a PktIO is opened with an +out_mode+ of
>> +ODP_PKTOUT_MODE_TM+.
>> > +
>> > +For TX processing via the Traffic Manager, applications use the
>> +odp_tm_enq()+
>> > +API:
>> > +[source,c]
>> > +-----
>> > +/** The odp_tm_enq() function is used to add packets to a given TM
>> system.
>> > + * Note that the System Metadata associated with the pkt needed by the
>> TM
>> > + * system is (a) a drop_eligible bit, (b) a two bit "pkt_color", (c) a
>> 16-bit
>> > + * pkt_len, and MAYBE? (d) a signed 8-bit shaper_len_adjust.
>> > + *
>> > + * If there is a non-zero shaper_len_adjust, then it is added to the
>> pkt_len
>> > + * after any non-zero shaper_len_adjust that is part of the shaper
>> profile.
>> > + *
>> > + * The pkt_color bits are a result of some earlier
>> Metering/Marking/Policing
>> > + * processing (typically ingress based), and should not be confused
>> with the
>> > + * shaper_color produced from the TM shaper entities within the
>> tm_inputs and
>> > + * tm_nodes.
>> > + *
>> > + * @param[in] tm_queue Specifies the tm_queue (and indirectly the TM
>> system).
>> > + * @param[in] pkt Handle to a packet.
>> > + * @return Returns 0 upon success, < 0 upon failure. One
>> of the
>> > + * more common failure reasons is WRED dropage.
>> > + */
>> > +int odp_tm_enq(odp_tm_queue_t tm_queue, odp_packet_t pkt);
>> > +-----
>> > +See the *Traffic Manager* section of this document for full
>> information about
>> > +Traffic Manager configuration and operation.
>> > diff --git a/doc/users-guide/users-guide.adoc
>> b/doc/users-guide/users-guide.adoc
>> > index d476225..ea24eaf 100644
>> > --- a/doc/users-guide/users-guide.adoc
>> > +++ b/doc/users-guide/users-guide.adoc
>> > @@ -1023,6 +1023,8 @@ a valid UDP packet. Applications are expected to
>> exercise appropriate care
>> > when changing packet metadata to ensure that the resulting metadata
>> changes
>> > reflect the actual changed packet structure that the application has
>> made.
>> >
>> > +include::users-guide-pktio.adoc[]
>> > +
>> > == Cryptographic services
>> >
>> > ODP provides support for cryptographic operations required by various
>> security
>> > --
>> > 2.5.0
>> >
>> > _______________________________________________
>> > lng-odp mailing list
>> > lng-odp@lists.linaro.org
>> > https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>> Interresting reading, and nice to see the user doc growing! Thanks!
>>
>
> Thanks for the careful review and questions.
>
>
>>
>> Christophe
>>
>
>
_______________________________________________
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp
