On 11 December 2015 at 10:08, Bill Fischofer <bill.fischo...@linaro.org>
wrote:
> The problem is that this is documentation text (pseudocode) not code and
> yet some tools insist on trying to treat it as if it's a code patch.
> Checkpatch should not apply to documentation this way.
>
In this case then we need to turn all the other spaces to tabs becasue now
we have a mix.
I want to keep it consistent one way or the other or reviewing will be
difficult depending on the viewers tab setting.
+...thread init processing
+
+while (1) {
+ odp_event_t ev; <-- only this one is newly tabbed
+ odp_queue_t which_q;
+ ev = odp_schedule(&which_q, <wait option>);
+ ...process the event
+}
+----
I honestly think it is just easier to not use tabs in the docs.
>
> On Fri, Dec 11, 2015 at 9:04 AM, Mike Holmes <mike.hol...@linaro.org>
> wrote:
>
>> Using patch:
>> lng-odp_PATCHv3_doc_userguide_add_application_programming_section.mbox
>> Trying to apply patch
>> Patch applied
>> Applying: doc: userguide: add application programming section
>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:446:
>> indent with spaces.
>> odp_event_t ev;
>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:613:
>> indent with spaces.
>> odp_init_local();
>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:616:
>> indent with spaces.
>> while (1) {
>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:617:
>> indent with spaces.
>> ev = odp_schedule(&which_q, ODP_SCHED_WAIT);
>> warning: 4 lines add whitespace errors.
>> 0001-doc-userguide-add-application-programming-section.patch: unified
>> diff output, UTF-8 Unicode text
>>
>> When I look at all these with git show I see a tab got in there rather
>> than a space which is inconsistent with the rest of the text.
>>
>> Do you have the .gitconfig whitespace =
>> trailing-space,space-before-tab,indent-with-non-tab,error-all if not you
>> may not be seeing some things.
>>
>>
>>
>>
>>
>> On 8 December 2015 at 17:36, Bill Fischofer <bill.fischo...@linaro.org>
>> wrote:
>>
>>> Complete the reformatting to standard asciidoc style, expand the
>>> ODP Application Programming section, and include a reorganized and
>>> expanded discussion of ODP queues.
>>>
>>> Signed-off-by: Bill Fischofer <bill.fischo...@linaro.org>
>>> ---
>>> doc/users-guide/users-guide.adoc | 451
>>> +++++++++++++++++++++++++++++++--------
>>> 1 file changed, 359 insertions(+), 92 deletions(-)
>>>
>>> diff --git a/doc/users-guide/users-guide.adoc
>>> b/doc/users-guide/users-guide.adoc
>>> index cf77fa0..d660fb8 100644
>>> --- a/doc/users-guide/users-guide.adoc
>>> +++ b/doc/users-guide/users-guide.adoc
>>> @@ -8,16 +8,19 @@ OpenDataPlane (ODP) Users-Guide
>>> Abstract
>>> --------
>>> This document is intended to guide a new ODP application developer.
>>> -Further details about ODP may be found at the http://opendataplane.org[ODP]
>>> home page.
>>> +Further details about ODP may be found at the http://opendataplane.org
>>> [ODP]
>>> +home page.
>>>
>>> .Overview of a system running ODP applications
>>> image::../images/overview.png[align="center"]
>>>
>>> -ODP is an API specification that allows many implementations to provide
>>> platform independence, automatic hardware acceleration and CPU scaling to
>>> high performance networking applications.
>>> -This document describes how to write an application that can
>>> successfully take advantage of the API.
>>> +ODP is an API specification that allows many implementations to provide
>>> +platform independence, automatic hardware acceleration and CPU scaling
>>> to
>>> +high performance networking applications. This document describes how
>>> to
>>> +write an application that can successfully take advantage of the API.
>>>
>>> :numbered:
>>> -== Introduction ==
>>> +== Introduction
>>> .OpenDataPlane Components
>>> image::../images/odp_components.png[align="center"]
>>>
>>> @@ -42,7 +45,7 @@ ODP API specification--that is the responsibility of
>>> each ODP implementation.
>>> * Application-centric. Covers functional needs of data plane
>>> applications.
>>> * Ensures portability by specifying the functional behavior of ODP.
>>> * Defined jointly and openly by application writers and platform
>>> implementers.
>>> -* Archiected to be implementable on a wide range of platforms
>>> efficiently
>>> +* Architected to be implementable on a wide range of platforms
>>> efficiently
>>> * Sponsored, governed, and maintained by the Linaro Networking Group
>>> (LNG)
>>>
>>> .ODP Implementations
>>> @@ -68,7 +71,7 @@ where the application will run on a target platform
>>> chosen by someone else.
>>> * One size does not fit all--supporting multiple implementations allows
>>> ODP
>>> to adapt to widely differing internals among platforms.
>>> * Anyone can create an ODP implementation tailored to their platform
>>> -* Distribution and mainteinance of each implementation is as owner
>>> wishes
>>> +* Distribution and maintenance of each implementation is as owner wishes
>>> - Open source or closed source as business needs determine
>>> - Have independent release cycles and service streams
>>> * Allows HW and SW innovation in how ODP APIs are implemented on each
>>> platform.
>>> @@ -100,7 +103,7 @@ drivers supported by DPDK.
>>> they are derived from a reference implementation.
>>>
>>> .ODP Validation Test Suite
>>> -Third, to enure consistency between different ODP implementations, ODP
>>> +Third, to ensure consistency between different ODP implementations, ODP
>>> consists of a validation suite that verifies that any given
>>> implementation of
>>> ODP faithfully provides the specified functional behavior of each ODP
>>> API.
>>> As a separate open source component, the validation suite may be used by
>>> @@ -115,16 +118,16 @@ ODP API specification.
>>> * Key to ensuring application portability across all ODP implementations
>>> * Tests that ODP implementations conform to the specified functional
>>> behavior
>>> of ODP APIs.
>>> -* Can be run at any time by users and vendors to validat implementations
>>> -od ODP.
>>> +* Can be run at any time by users and vendors to validate
>>> implementations
>>> +of ODP.
>>>
>>> -=== ODP API Specification Versioning ===
>>> +=== ODP API Specification Versioning
>>> As an evolving standard, the ODP API specification is released under an
>>> incrementing version number, and corresponding implementations of ODP,
>>> as well
>>> as the validation suite that verifies API conformance, are linked to
>>> this
>>> -version number. ODP versions are specified using a stanard three-level
>>> +version number. ODP versions are specified using a standard three-level
>>> number (major.minor.fixlevel) that are incremented according to the
>>> degree of
>>> -change the level represents. Increments to the fixlevel represent
>>> clarification
>>> +change the level represents. Increments to the fix level represent
>>> clarification
>>> of the specification or other minor changes that do not affect either
>>> the
>>> syntax or semantics of the specification. Such changes in the API
>>> specification
>>> are expected to be rare. Increments to the minor level
>>> @@ -136,26 +139,26 @@ the major level represent significant structural
>>> changes that most likely
>>> require some level of application source code change, again as
>>> documented in
>>> the release notes for that version.
>>>
>>> -=== ODP Implementation Versioning ===
>>> +=== ODP Implementation Versioning
>>> ODP implementations are free to use whatever release naming/numbering
>>> conventions they wish, as long as it is clear what level of the ODP API
>>> a given
>>> release implements. A recommended convention is to use the same three
>>> level
>>> numbering scheme where the major and minor numbers correspond to the
>>> ODP API
>>> -level and the fixlevel represents an implementation-defined service
>>> level
>>> +level and the fix level represents an implementation-defined service
>>> level
>>> associated with that API level implementation. The LNG-supplied ODP
>>> reference
>>> implementations follow this convention.
>>>
>>> -=== ODP Validation Test Suite Versioning ===
>>> +=== ODP Validation Test Suite Versioning
>>> The ODP validation test suite follows these same naming conventions.
>>> The major
>>> and minor release numbers correspond to the ODP API level that the suite
>>> -validates and the fixlevel represents the service level of the
>>> validation
>>> +validates and the fix level represents the service level of the
>>> validation
>>> suite itself for that API level.
>>>
>>> -=== ODP Design Goals ===
>>> +=== ODP Design Goals
>>> ODP has three primary goals that follow from its component structure.
>>> The first
>>> is application portability across a wide range of platforms. These
>>> platforms
>>> differ in terms of processor instruction set architecture, number and
>>> types of
>>> -application processing cores, memory oranization, as well as the number
>>> and
>>> +application processing cores, memory organization, as well as the
>>> number and
>>> type of platform specific hardware acceleration and offload features
>>> that
>>> are available. ODP applications can move from one conforming
>>> implementation
>>> to another with at most a recompile.
>>> @@ -175,7 +178,7 @@ of processing cores that are available to realize
>>> application function. The
>>> result is that an application written to this model does not require
>>> redesign
>>> as it scales from 4, to 40, to 400 cores.
>>>
>>> -== Organization of this Document ==
>>> +== Organization of this Document
>>> This document is organized into several sections. The first presents a
>>> high
>>> level overview of the ODP API component areas and their associated
>>> abstract
>>> data types. This section introduces ODP APIs at a conceptual level.
>>> @@ -190,14 +193,14 @@ full reference specification for each API. The
>>> latter is intended to be used
>>> by ODP application programmers, as well as implementers, to understand
>>> the
>>> precise syntax and semantics of each API.
>>>
>>> -== ODP API Concepts ==
>>> +== ODP API Concepts
>>> ODP programs are built around several conceptual structures that every
>>> -appliation programmer needs to be familiar with to use ODP effectively.
>>> The
>>> +application programmer needs to be familiar with to use ODP
>>> effectively. The
>>> main ODP concepts are:
>>> Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, Timer,
>>> and Synchronizer.
>>>
>>> -=== Thread ===
>>> +=== Thread
>>> The thread is the fundamental programming unit in ODP. ODP
>>> applications are
>>> organized into a collection of threads that perform the work that the
>>> application is designed to do. ODP threads may or may not share memory
>>> with
>>> @@ -209,7 +212,7 @@ A control thread is a supervisory thread that
>>> organizes
>>> the operation of worker threads. Worker threads, by contrast, exist to
>>> perform the main processing logic of the application and employ a run to
>>> completion model. Worker threads, in particular, are intended to
>>> operate on
>>> -dedicated processing cores, especially in many core proessing
>>> environments,
>>> +dedicated processing cores, especially in many core processing
>>> environments,
>>> however a given implementation may multitask multiple threads on a
>>> single
>>> core if desired (typically on smaller and lower performance target
>>> environments).
>>> @@ -219,7 +222,7 @@ _thread mask_ and _scheduler group_ that determine
>>> where they can run and
>>> the type of work that they can handle. These will be discussed in
>>> greater
>>> detail later.
>>>
>>> -=== Event ===
>>> +=== Event
>>> Events are what threads process to perform their work. Events can
>>> represent
>>> new work, such as the arrival of a packet that needs to be processed,
>>> or they
>>> can represent the completion of requests that have executed
>>> asynchronously.
>>> @@ -232,7 +235,7 @@ References to events are via handles of abstract
>>> type +odp_event_t+. Cast
>>> functions are provided to convert these into specific handles of the
>>> appropriate type represented by the event.
>>>
>>> -=== Queue ===
>>> +=== Queue
>>> A queue is a message passing channel that holds events. Events can be
>>> added to a queue via enqueue operations or removed from a queue via
>>> dequeue
>>> operations. The endpoints of a queue will vary depending on how it is
>>> used.
>>> @@ -244,7 +247,7 @@ stateful processing on events as well as stateless
>>> processing.
>>>
>>> Queues are represented by handles of abstract type +odp_queue_t+.
>>>
>>> -=== Pool ===
>>> +=== Pool
>>> A pool is a shared memory area from which elements may be drawn. Pools
>>> represent the backing store for events, among other things. Pools are
>>> typically created and destroyed by the application during
>>> initialization and
>>> @@ -256,32 +259,32 @@ are Buffer and Packet.
>>>
>>> Pools are represented by handles of abstract type +odp_pool_t+.
>>>
>>> -=== Shared Memory ===
>>> +=== Shared Memory
>>> Shared memory represents raw blocks of storage that are sharable between
>>> threads. They are the building blocks of pools but can be used directly
>>> by
>>> ODP applications if desired.
>>>
>>> Shared memory is represented by handles of abstract type +odp_shm_t+.
>>>
>>> -=== Buffer ===
>>> +=== Buffer
>>> A buffer is a fixed sized block of shared storage that is used by ODP
>>> components and/or applications to realize their function. Buffers
>>> contain
>>> zero or more bytes of application data as well as system maintained
>>> metadata that provide information about the buffer, such as its size or
>>> the
>>> pool it was allocated from. Metadata is an important ODP concept
>>> because it
>>> allows for arbitrary amounts of side information to be associated with
>>> an
>>> -ODP object. Most ODP objects have assocaited metadata and this metadata
>>> is
>>> +ODP object. Most ODP objects have associated metadata and this metadata
>>> is
>>> manipulated via accessor functions that act as getters and setters for
>>> -this information. Getter acces functions permit an application to read
>>> +this information. Getter access functions permit an application to read
>>> a metadata item, while setter access functions permit an application to
>>> write
>>> a metadata item. Note that some metadata is inherently read only and
>>> thus
>>> no setter is provided to manipulate it. When object have multiple
>>> metadata
>>> items, each has its own associated getter and/or setter access function
>>> to
>>> inspect or manipulate it.
>>>
>>> -Buffers are represened by handles of abstract type +odp_buffer_t+.
>>> +Buffers are represented by handles of abstract type +odp_buffer_t+.
>>>
>>> -=== Packet ===
>>> +=== Packet
>>> Packets are received and transmitted via I/O interfaces and represent
>>> the basic data that data plane applications manipulate.
>>> Packets are drawn from pools of type +ODP_POOL_PACKET+.
>>> @@ -294,7 +297,7 @@ with each packet for its own use.
>>>
>>> Packets are represented by handles of abstract type +odp_packet_t+.
>>>
>>> -=== PktIO ===
>>> +=== PktIO
>>> PktIO is how ODP represents I/O interfaces. A pktio object is a logical
>>> port capable of receiving and/or transmitting packets. This may be
>>> directly
>>> supported by the underlying platform as an integrated feature,
>>> @@ -302,7 +305,7 @@ or may represent a device attached via a PCIE or
>>> other bus.
>>>
>>> PktIOs are represented by handles of abstract type +odp_pktio_t+.
>>>
>>> -=== Timer ===
>>> +=== Timer
>>> Timers are how ODP applications measure and respond to the passage of
>>> time.
>>> Timers are drawn from specialized pools called timer pools that have
>>> their
>>> own abstract type (+odp_timer_pool_t+). Applications may have many
>>> timers
>>> @@ -310,7 +313,7 @@ active at the same time and can set them to use
>>> either relative or absolute
>>> time. When timers expire they create events of type +odp_timeout_t+,
>>> which
>>> serve as notifications of timer expiration.
>>>
>>> -=== Synchronizer ===
>>> +=== Synchronizer
>>> Multiple threads operating in parallel typically require various
>>> synchronization services to permit them to operate in a reliable and
>>> coordinated manner. ODP provides a rich set of locks, barriers, and
>>> similar
>>> @@ -325,7 +328,7 @@ flow of work through an ODP application. These
>>> include the Classifier,
>>> Scheduler, and Traffic Manager. These components relate to the three
>>> main stages of packet processing: Receive, Process, and Transmit.
>>>
>>> -=== Classifier ===
>>> +=== Classifier
>>> The *Classifier* provides a suite of APIs that control packet receive
>>> (RX)
>>> processing.
>>>
>>> @@ -362,8 +365,8 @@ Note that the use of the classifier is optional.
>>> Applications may directly
>>> receive packets from a corresponding PktIO input queue via direct
>>> polling
>>> if they choose.
>>>
>>> -=== Scheduler ===
>>> -The *Scheduler* provides a suite of APIs that control scalabable event
>>> +=== Scheduler
>>> +The *Scheduler* provides a suite of APIs that control scalable event
>>> processing.
>>>
>>> .ODP Scheduler and Event Processing
>>> @@ -391,10 +394,10 @@ scheduled back to a thread to continue processing
>>> with the results of the
>>> requested asynchronous operation.
>>>
>>> Threads themselves can enqueue events to queues for downstream
>>> processing
>>> -by other threads, permitting flexibility in how applicaitions structure
>>> +by other threads, permitting flexibility in how applications structure
>>> themselves to maximize concurrency.
>>>
>>> -=== Traffic Manager ===
>>> +=== Traffic Manager
>>> The *Traffic Manager* provides a suite of APIs that control traffic
>>> shaping and
>>> Quality of Service (QoS) processing for packet output.
>>>
>>> @@ -413,23 +416,33 @@ goals. Again, the advantage here is that on many
>>> platforms traffic management
>>> functions are implemented in hardware, permitting transparent offload of
>>> this work.
>>>
>>> -Glossary
>>> ---------
>>> -[glossary]
>>> -odp_worker::
>>> - An opd_worker is a type of odp_thread. It will usually be isolated
>>> from the scheduling of any host operating system and is intended for
>>> fast-path processing with a low and predictable latency. Odp_workers will
>>> not generally receive interrupts and will run to completion.
>>> -odp_control::
>>> - An odp_control is a type of odp_thread. It will be isolated from
>>> the host operating system house keeping tasks but will be scheduled by it
>>> and may receive interrupts.
>>> -odp_thread::
>>> - An odp_thread is a flow of execution that in a Linux environment
>>> could be a Linux process or thread.
>>> -event::
>>> - An event is a notification that can be placed in a queue.
>>> -
>>> -The include structure
>>> ----------------------
>>> -Applications only include the 'include/odp.h file which includes the
>>> 'platform/<implementation name>/include/plat' files to provide a complete
>>> definition of the API on that platform.
>>> -The doxygen documentation defining the behavior of the ODP API is all
>>> contained in the public API files, and the actual definitions for an
>>> implementation will be found in the per platform directories.
>>> -Per-platform data that might normally be a #define can be recovered via
>>> the appropriate access function if the #define is not directly visible to
>>> the application.
>>> +== ODP Application Programming
>>> +At the highest level, an *ODP Application* is a program that uses one
>>> or more
>>> +ODP APIs. Because ODP is a framework rather than a programming
>>> environment,
>>> +applications are free to also use other APIs that may or may not
>>> provide the
>>> +same portability characteristics as ODP APIs.
>>> +
>>> +ODP applications vary in terms of what they do and how they operate,
>>> but in
>>> +general all share the following characteristics:
>>> +
>>> +. They are organized into one or more _threads_ that execute in
>>> parallel.
>>> +. These threads communicate and coordinate their activities using
>>> various
>>> +_synchronization_ mechanisms.
>>> +. They receive packets from one or more _packet I/O interfaces_.
>>> +. They examine, transform, or otherwise process packets.
>>> +. They transmit packets to one or more _packet I/O interfaces_.
>>> +
>>> +ODP provides APIs to assist in each of these areas.
>>> +
>>> +=== The include structure
>>> +Applications only include the 'include/odp.h' file, which includes the
>>> +'platform/<implementation name>/include/odp' files to provide a complete
>>> +definition of the API on that platform. The doxygen documentation
>>> defining
>>> +the behavior of the ODP API is all contained in the public API files,
>>> and the
>>> +actual definitions for an implementation will be found in the per
>>> platform
>>> +directories. Per-platform data that might normally be a +#define+ can be
>>> +recovered via the appropriate access function if the #define is not
>>> directly
>>> +visible to the application.
>>>
>>> .Users include structure
>>> ----
>>> @@ -442,51 +455,305 @@ Per-platform data that might normally be a
>>> #define can be recovered via the appr
>>> │ └── odp.h This file should be the only file included by the
>>> application.
>>> ----
>>>
>>> -Initialization
>>> ---------------
>>> -IMPORTANT: ODP depends on the application to perform a graceful
>>> shutdown, calling the terminate functions should only be done when the
>>> application is sure it has closed the ingress and subsequently drained all
>>> queues etc.
>>> +=== Initialization
>>> +IMPORTANT: ODP depends on the application to perform a graceful
>>> shutdown,
>>> +calling the terminate functions should only be done when the
>>> application is
>>> +sure it has closed the ingress and subsequently drained all queues, etc.
>>> +
>>> +=== Startup
>>> +The first API that must be called by an ODP application is
>>> 'odp_init_global()'.
>>> +This takes two pointers. The first, +odp_init_t+, contains ODP
>>> initialization
>>> +data that is platform independent and portable, while the second,
>>> ++odp_platform_init_t+, is passed unparsed to the implementation
>>> +to be used for platform specific data that is not yet, or may never be
>>> +suitable for the ODP API.
>>> +
>>> +Calling odp_init_global() establishes the ODP API framework and MUST be
>>> +called before any other ODP API may be called. Note that it is only
>>> called
>>> +once per application. Following global initialization, each thread in
>>> turn
>>> +calls 'odp_init_local()' is called. This establishes the local ODP
>>> thread
>>> +context for that thread and MUST be called before other ODP APIs may be
>>> +called by that thread.
>>> +
>>> +=== Shutdown
>>> +Shutdown is the logical reverse of the initialization procedure, with
>>> +'odp_term_local()' called for each thread before 'odp_term_global()' is
>>> +called to terminate ODP.
>>> +
>>> +.ODP Application Structure Flow Diagram
>>> +image::../images/resource_management.png[align="center"]
>>>
>>> -Startup
>>> -~~~~~~~~
>>> -The first API that must be called is 'odp_init_global()'.
>>> -This takes two pointers, odp_init_t contains ODP initialization data
>>> that is platform independent and portable.
>>> -The second odp_platform_init_t is passed un parsed to the
>>> implementation and can be used for platform specific data that is not yet,
>>> or may never be suitable for the ODP API.
>>> +== Common Conventions
>>> +Many ODP APIs share common conventions regarding their arguments and
>>> return
>>> +types. This section highlights some of the more common and frequently
>>> used
>>> +conventions.
>>> +
>>> +=== Handles and Special Designators
>>> +ODP resources are represented via _handles_ that have abstract type
>>> +_odp_resource_t_. So pools are represented by handles of type
>>> +odp_pool_t+,
>>> +queues by handles of type +odp_queue_t+, etc. Each such type
>>> +has a distinguished type _ODP_RESOURCE_INVALID_ that is used to
>>> indicate a
>>> +handle that does not refer to a valid resource of that type. Resources
>>> are
>>> +typically created via an API named _odp_resource_create()_ that returns
>>> a
>>> +handle of type _odp_resource_t_ that represents the created object. This
>>> +returned handle is set to _ODP_RESOURCE_INVALID_ if, for example, the
>>> +resource could not be created due to resource exhaustion. Invalid
>>> resources
>>> +do not necessarily represent error conditions. For example,
>>> +ODP_EVENT_INVALID+
>>> +in response to an +odp_queue_deq()+ call to get an event from a queue
>>> simply
>>> +indicates that the queue is empty.
>>> +
>>> +=== Addressing Scope
>>> +Unless specifically noted in the API, all ODP resources are global to
>>> the ODP
>>> +application, whether it runs as a single process or multiple processes.
>>> ODP
>>> +handles therefore have common meaning within an ODP application but
>>> have no
>>> +meaning outside the scope of the application.
>>> +
>>> +=== Resources and Names
>>> +Many ODP resource objects, such as pools and queues, support an
>>> +application-specified character string _name_ that is associated with
>>> an ODP
>>> +object at create time. This name serves two purposes: documentation,
>>> and
>>> +lookup. The lookup function is particularly useful to allow an ODP
>>> application
>>> +that is divided into multiple processes to obtain the handle for the
>>> common
>>> +resource.
>>> +
>>> +== Queues
>>> +Queues are the fundamental event sequencing mechanism provided by ODP
>>> and all
>>> +ODP applications make use of them either explicitly or implicitly.
>>> Queues are
>>> +created via the 'odp_queue_create()' API that returns a handle of type
>>> ++odp_queue_t+ that is used to refer to this queue in all subsequent
>>> APIs that
>>> +reference it. Queues have one of two ODP-defined _types_, POLL, and
>>> SCHED that
>>> +determine how they are used. POLL queues directly managed by the ODP
>>> +application while SCHED queues make use of the *ODP scheduler* to
>>> provide
>>> +automatic scalable dispatching and synchronization services.
>>> +
>>> +.Operations on POLL queues
>>> +[source,c]
>>> +----
>>> +odp_queue_t poll_q1 = odp_queue_create("poll queue 1",
>>> ODP_QUEUE_TYPE_POLL, NULL);
>>> +odp_queue_t poll_q2 = odp_queue_create("poll queue 2",
>>> ODP_QUEUE_TYPE_POLL, NULL);
>>> +...
>>> +odp_event_t ev = odp_queue_deq(poll_q1);
>>> +...do something
>>> +int rc = odp_queue_enq(poll_q2, ev);
>>> +----
>>>
>>> -The second API that must be called is 'odp_init_local()', this must be
>>> called once per odp_thread, regardless of odp_thread type. Odp_threads may
>>> be of type ODP_THREAD_WORKER or ODP_THREAD_CONTROL
>>> +The key distinction is that dequeueing events from POLL queues is an
>>> +application responsibility while dequeueing events from SCHED queues is
>>> the
>>> +responsibility of the ODP scheduler.
>>>
>>> -Shutdown
>>> -~~~~~~~~~
>>> -Shutdown is the logical reverse of the initialization procedure, with
>>> 'odp_thread_term()' called for each worker before 'odp_term_global()' is
>>> called.
>>> +.Operations on SCHED queues
>>> +[source,c]
>>> +----
>>> +odp_queue_param_t qp;
>>> +odp_queue_param_init(&qp);
>>> +odp_schedule_prio_t prio = ...;
>>> +odp_schedule_group_t sched_group = ...;
>>> +qp.sched.prio = prio;
>>> +qp.sched.sync = ODP_SCHED_SYNC_[NONE|ATOMIC|ORDERED];
>>> +qp.sched.group = sched_group;
>>> +qp.lock_count = n; /* Only relevant for ordered queues */
>>> +odp_queue_t sched_q1 = odp_queue_create("sched queue 1",
>>> ODP_QUEUE_TYPE_SCHED, &qp);
>>> +
>>> +...thread init processing
>>> +
>>> +while (1) {
>>> + odp_event_t ev;
>>> + odp_queue_t which_q;
>>> + ev = odp_schedule(&which_q, <wait option>);
>>> + ...process the event
>>> +}
>>> +----
>>>
>>> -image::../images/resource_management.png[align="center"]
>>> +With scheduled queues, events are sent to a queue, and the the sender
>>> chooses
>>> +a queue based on the service it needs. The sender does not need to know
>>> +which ODP thread (on which core) or hardware accelerator will process
>>> +the event, but all the events on a queue are eventually scheduled and
>>> processed.
>>> +
>>> +As can be seen, SCHED queues have additional attributes that are
>>> specified at
>>> +queue create that control how the scheduler is to process events
>>> contained
>>> +on them. These include group, priority, and synchronization class.
>>> +
>>> +=== Scheduler Groups
>>> +The scheduler's dispatching job is to return the next event from the
>>> highest
>>> +priority SCHED queue that the caller is eligible to receive events from.
>>> +This latter consideration is determined by the queues _scheduler
>>> group_, which
>>> +is set at queue create time, and by the caller's _scheduler group mask_
>>> that
>>> +indicates which scheduler group(s) it belongs to. Scheduler groups are
>>> +represented by handles of type +odp_scheduler_group_t+ and are created
>>> by
>>> +the *odp_scheduler_group_create()* API. A number of scheduler groups are
>>> +_predefined_ by ODP. These include +ODP_SCHED_GROUP_ALL+ (all threads),
>>> ++ODP_SCHED_GROUP_WORKER+ (all worker threads), and
>>> +ODP_SCHED_GROUP_CONTROL+
>>> +(all control threads). The application is free to create additional
>>> scheduler
>>> +groups for its own purpose and threads can join or leave scheduler
>>> groups
>>> +using the *odp_scheduler_group_join()* and
>>> *odp_scheduler_group_leave()* APIs
>>> +
>>> +=== Scheduler Priority
>>> +The +prio+ field of the +odp_queue_param_t+ specifies the queue's
>>> scheduling
>>> +priority, which is how queues within eligible scheduler groups are
>>> selected
>>> +for dispatch. Queues have a default scheduling priority of NORMAL but
>>> can be
>>> +set to HIGHEST or LOWEST according to application needs.
>>> +
>>> +=== Scheduler Synchronization
>>> +In addition to its dispatching function, which provide automatic
>>> scalability to
>>> +ODP applications in many core environments, the other main function of
>>> the
>>> +scheduler is to provide event synchronization services that greatly
>>> simplify
>>> +application programming in a parallel processing environment. A queue's
>>> +SYNC mode determines how the scheduler handles the synchronization
>>> processing
>>> +of multiple events originating from the same queue.
>>> +
>>> +Three types of queue scheduler synchronization area supported: Parallel,
>>> +Atomic, and Ordered.
>>> +
>>> +==== Parallel Queues
>>> +SCHED queues that specify a sync mode of ODP_SCHED_SYNC_NONE are
>>> unrestricted
>>> +in how events are processed.
>>> +
>>> +.Parallel Queue Scheduling
>>> +image::../images/parallel_queue.png[align="center"]
>>>
>>> -Queues
>>> -------
>>> -There are three queue types, atomic, ordered and parallel.
>>> -A queue belongs to a single odp_worker and a odp_worker may have
>>> multiple queues.
>>> +All events held on parallel queues are eligible to be scheduled
>>> simultaneously
>>> +and any required synchronization between them is the responsibility of
>>> the
>>> +application. Events originating from parallel queues thus have the
>>> highest
>>> +throughput rate, however they also potentially involve the most work on
>>> the
>>> +part of the application. In the Figure above, four threads are calling
>>> +*odp_schedule()* to obtain events to process. The scheduler has assigned
>>> +three events from the first queue to three threads in parallel. The
>>> fourth
>>> +thread is processing a single event from the third queue. The second
>>> queue
>>> +might either be empty, of lower priority, or not in a scheduler group
>>> matching
>>> +any of the threads being serviced by the scheduler.
>>> +
>>> +=== Atomic Queues
>>> +Atomic queues simplify event synchronization because only a single event
>>> +from a given atomic queue may be processed at a time. Events scheduled
>>> from
>>> +atomic queues thus can be processed lock free because the locking is
>>> being
>>> +done implicitly by the scheduler.
>>> +
>>> +.Atomic Queue Scheduling
>>> +image::../images/atomic_queue.png[align="center"]
>>>
>>> -Events are sent to a queue, and the the sender chooses a queue based on
>>> the service it needs.
>>> -The sender does not need to know which odp_worker (on which core) or HW
>>> accelerator will process the event, but all the events on a queue are
>>> eventually scheduled and processed.
>>> +In this example, no matter how many events may be held in an atomic
>>> queue, only
>>> +one of them can be scheduled at a time. Here two threads process events
>>> from
>>> +two different atomic queues. Note that there is no synchronization
>>> between
>>> +different atomic queues, only between events originating from the same
>>> atomic
>>> +queue. The queue context associated with the atomic queue is held until
>>> the
>>> +next call to the scheduler or until the application explicitly releases
>>> it
>>> +via a call to *odp_schedule_release_atomic()*.
>>>
>>> -NOTE: Both ordered and parallel queue types improve throughput over an
>>> atomic queue (due to parallel event processing), but the user has to take
>>> care of the context data synchronization (if needed).
>>> +Note that while atomic queues simplify programming, the serial nature of
>>> +atomic queues will impair scaling.
>>>
>>> -Atomic Queue
>>> -~~~~~~~~~~~~
>>> -Only one event at a time may be processed from a given queue. The
>>> processing maintains order and context data synchronization but this will
>>> impair scaling.
>>> +=== Ordered Queues
>>> +Ordered queues provide the best of both worlds by providing the inherent
>>> +scaleabilty of parallel queues, with the easy synchronization of atomic
>>> +queues.
>>>
>>> -.Overview Atomic Queue processing
>>> -image::../images/atomic_queue.png[align="center"]
>>> +.Ordered Queue Scheduling
>>> +image::../images/ordered_queue.png[align="center"]
>>>
>>> -Ordered Queue
>>> -~~~~~~~~~~~~~
>>> -An ordered queue will ensure that the sequencing at the output is
>>> identical to that of the input, but multiple events may be processed
>>> simultaneously and the order is restored before the events move to the next
>>> queue
>>> +When scheduling events from an ordered queue, the scheduler dispatches
>>> multiple
>>> +events from the queue in parallel to different threads, however the
>>> scheduler
>>> +also ensures that the relative sequence of these events on output queues
>>> +is identical to their sequence from their originating ordered queue.
>>> +
>>> +As with atomic queues, the ordering guarantees associated with ordered
>>> queues
>>> +refer to events originating from the same queue, not for those
>>> originating on
>>> +different queues. Thus in this figure three thread are processing
>>> events 5, 3,
>>> +and 4, respectively from the first ordered queue. Regardless of how
>>> these
>>> +threads complete processing, these events will appear in their original
>>> +relative order on their output queue.
>>> +
>>> +==== Order Preservation
>>> +Relative order is preserved independent of whether events are being
>>> sent to
>>> +different output queues. For example, if some events are sent to
>>> output queue
>>> +A while others are sent to output queue B then the events on these
>>> output
>>> +queues will still be in the same relative order as they were on their
>>> +originating queue. Similarly, if the processing consumes events so
>>> that no
>>> +output is issued for some of them (_e.g.,_ as part of IP fragment
>>> reassembly
>>> +processing) then other events will still be correctly ordered with
>>> respect to
>>> +these sequence gaps. Finally, if multiple events are enqueued for a
>>> given
>>> +order (_e.g.,_ as part of packet segmentation processing for MTU
>>> +considerations), then each of these events will occupy the originator's
>>> +sequence in the target output queue(s). In this case the relative order
>>> of these
>>> +events will be in the order that the thread issued *odp_queue_enq()*
>>> calls for
>>> +them.
>>> +
>>> +The ordered context associated with the dispatch of an event from an
>>> ordered
>>> +queue lasts until the next scheduler call or until explicitly released
>>> by
>>> +the thread calling *odp_schedule_release_ordered()*. This call may be
>>> used
>>> +as a performance advisory that the thread no longer requires ordering
>>> +guarantees for the current context. As a result, any subsequent enqueues
>>> +within the current scheduler context will be treated as if the thread
>>> was
>>> +operating in a parallel queue context.
>>> +
>>> +==== Ordered Locking
>>> +Another powerful feature of the scheduler's handling of ordered queues
>>> is
>>> +*ordered locks*. Each ordered queue has associated with it a number of
>>> ordered
>>> +locks as specified by the _lock_count_ parameter at queue create time.
>>> +
>>> +Ordered locks provide an efficient means to perform in-order sequential
>>> +processing within an ordered context. For example, supposed events with
>>> relative
>>> +order 5, 6, and 7 are executing in parallel by three different threads.
>>> An
>>> +ordered lock will enable these threads to synchronize such that they can
>>> +perform some critical section in their originating queue order. The
>>> number of
>>> +ordered locks supported for each ordered queue is implementation
>>> dependent (and
>>> +queryable via the *odp_config_max_ordered_locks_per_queue()* API). If
>>> the
>>> +implementation supports multiple ordered locks then these may be used to
>>> +protect different ordered critical sections within a given ordered
>>> context.
>>> +
>>> +==== Summary: Ordered Queues
>>> +To see how these considerations fit together, consider the following
>>> code:
>>> +
>>> +.Processing with Ordered Queues
>>> +[source,c]
>>> +----
>>> +void worker_thread()
>>> +{
>>> + odp_init_local();
>>> + ...other initialization processing
>>> +
>>> + while (1) {
>>> + ev = odp_schedule(&which_q, ODP_SCHED_WAIT);
>>> + ...process events in parallel
>>> + odp_schedule_order_lock(0);
>>> + ...critical section processed in order
>>> + odp_schedule_order_unlock(0);
>>> + ...continue processing in parallel
>>> + odp_queue_enq(dest_q, ev);
>>> + }
>>> +}
>>> +----
>>>
>>> -.Overview Ordered Queue processing
>>> -image::../images/ordered_queue.png[align="center"]
>>> +This represents a simplified structure for a typical worker thread
>>> operating
>>> +on ordered queues. Multiple events are processed in parallel and the
>>> use of
>>> +ordered queues ensures that they will be placed on +dest_q+ in the same
>>> order
>>> +as they originated. While processing in parallel, the use of ordered
>>> locks
>>> +enables critical sections to be processed in order within the overall
>>> parallel
>>> +flow. When a thread arrives at the _odp_schedule_order_lock()_ call, it
>>> waits
>>> +until the locking order for this lock for all prior events has been
>>> resolved
>>> +and then enters the critical section. The _odp_schedule_order_unlock()_
>>> call
>>> +releases the critical section and allows the next order to enter it.
>>>
>>> -Parallel Queue
>>> -~~~~~~~~~~~~~~
>>> -There are no restrictions on the number of events being processed.
>>> +=== Queue Scheduling Summary
>>>
>>> -.Overview parallel Queue processing
>>> -image::../images/parallel_queue.png[align="center"]
>>> +NOTE: Both ordered and parallel queues improve throughput over atomic
>>> queues
>>> +due to parallel event processing, but require that the application take
>>> +steps to ensure context data synchronization if needed.
>>> +
>>> +== Glossary
>>> +[glossary]
>>> +worker thread::
>>> + A worker is a type of ODP thread. It will usually be isolated from
>>> + the scheduling of any host operating system and is intended for
>>> fast-path
>>> + processing with a low and predictable latency. Worker threads will
>>> not
>>> + generally receive interrupts and will run to completion.
>>> +control thread::
>>> + A control threadis a type of ODP thread. It will be isolated from
>>> the host
>>> + operating system house keeping tasks but will be scheduled by it
>>> and may
>>> + receive interrupts.
>>> +thread::
>>> + An ODP thread is a flow of execution that in a Linux environment
>>> could be
>>> + a Linux process or thread.
>>> +event::
>>> + An event is a notification that can be placed in a queue.
>>> +queue::
>>> + A communication channel that holds events
>>> --
>>> 2.1.4
>>>
>>> _______________________________________________
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>
>>
>>
>> --
>> Mike Holmes
>> Technical Manager - Linaro Networking Group
>> Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
>>
>>
>>
>
--
Mike Holmes
Technical Manager - Linaro Networking Group
Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
_______________________________________________
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp
