The checkpatch warning is not present with the CI like version of Ubuntu [1], I see that on 15.04 it is not a checkpatch issue.
[1] https://hub.docker.com/r/roxell/check-odp-ubuntu/ On 11 December 2015 at 10:18, Mike Holmes <mike.hol...@linaro.org> wrote: > > > On 11 December 2015 at 10:08, Bill Fischofer <bill.fischo...@linaro.org> > wrote: > >> The problem is that this is documentation text (pseudocode) not code and >> yet some tools insist on trying to treat it as if it's a code patch. >> Checkpatch should not apply to documentation this way. >> > > In this case then we need to turn all the other spaces to tabs becasue now > we have a mix. > I want to keep it consistent one way or the other or reviewing will be > difficult depending on the viewers tab setting. > > +...thread init processing > + > +while (1) { > + odp_event_t ev; <-- only this one is newly tabbed > + odp_queue_t which_q; > + ev = odp_schedule(&which_q, <wait option>); > + ...process the event > +} > +---- > > > I honestly think it is just easier to not use tabs in the docs. > > > >> >> On Fri, Dec 11, 2015 at 9:04 AM, Mike Holmes <mike.hol...@linaro.org> >> wrote: >> >>> Using patch: >>> lng-odp_PATCHv3_doc_userguide_add_application_programming_section.mbox >>> Trying to apply patch >>> Patch applied >>> Applying: doc: userguide: add application programming section >>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:446: >>> indent with spaces. >>> odp_event_t ev; >>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:613: >>> indent with spaces. >>> odp_init_local(); >>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:616: >>> indent with spaces. >>> while (1) { >>> /home/mike/git/check-odp/build/odp-apply/.git/rebase-apply/patch:617: >>> indent with spaces. >>> ev = odp_schedule(&which_q, ODP_SCHED_WAIT); >>> warning: 4 lines add whitespace errors. >>> 0001-doc-userguide-add-application-programming-section.patch: unified >>> diff output, UTF-8 Unicode text >>> >>> When I look at all these with git show I see a tab got in there rather >>> than a space which is inconsistent with the rest of the text. >>> >>> Do you have the .gitconfig whitespace = >>> trailing-space,space-before-tab,indent-with-non-tab,error-all if not you >>> may not be seeing some things. >>> >>> >>> >>> >>> >>> On 8 December 2015 at 17:36, Bill Fischofer <bill.fischo...@linaro.org> >>> wrote: >>> >>>> Complete the reformatting to standard asciidoc style, expand the >>>> ODP Application Programming section, and include a reorganized and >>>> expanded discussion of ODP queues. >>>> >>>> Signed-off-by: Bill Fischofer <bill.fischo...@linaro.org> >>>> --- >>>> doc/users-guide/users-guide.adoc | 451 >>>> +++++++++++++++++++++++++++++++-------- >>>> 1 file changed, 359 insertions(+), 92 deletions(-) >>>> >>>> diff --git a/doc/users-guide/users-guide.adoc >>>> b/doc/users-guide/users-guide.adoc >>>> index cf77fa0..d660fb8 100644 >>>> --- a/doc/users-guide/users-guide.adoc >>>> +++ b/doc/users-guide/users-guide.adoc >>>> @@ -8,16 +8,19 @@ OpenDataPlane (ODP) Users-Guide >>>> Abstract >>>> -------- >>>> This document is intended to guide a new ODP application developer. >>>> -Further details about ODP may be found at the >>>> http://opendataplane.org[ODP] >>>> home page. >>>> +Further details about ODP may be found at the http://opendataplane.org >>>> [ODP] >>>> +home page. >>>> >>>> .Overview of a system running ODP applications >>>> image::../images/overview.png[align="center"] >>>> >>>> -ODP is an API specification that allows many implementations to >>>> provide platform independence, automatic hardware acceleration and CPU >>>> scaling to high performance networking applications. >>>> -This document describes how to write an application that can >>>> successfully take advantage of the API. >>>> +ODP is an API specification that allows many implementations to provide >>>> +platform independence, automatic hardware acceleration and CPU scaling >>>> to >>>> +high performance networking applications. This document describes how >>>> to >>>> +write an application that can successfully take advantage of the API. >>>> >>>> :numbered: >>>> -== Introduction == >>>> +== Introduction >>>> .OpenDataPlane Components >>>> image::../images/odp_components.png[align="center"] >>>> >>>> @@ -42,7 +45,7 @@ ODP API specification--that is the responsibility of >>>> each ODP implementation. >>>> * Application-centric. Covers functional needs of data plane >>>> applications. >>>> * Ensures portability by specifying the functional behavior of ODP. >>>> * Defined jointly and openly by application writers and platform >>>> implementers. >>>> -* Archiected to be implementable on a wide range of platforms >>>> efficiently >>>> +* Architected to be implementable on a wide range of platforms >>>> efficiently >>>> * Sponsored, governed, and maintained by the Linaro Networking Group >>>> (LNG) >>>> >>>> .ODP Implementations >>>> @@ -68,7 +71,7 @@ where the application will run on a target platform >>>> chosen by someone else. >>>> * One size does not fit all--supporting multiple implementations >>>> allows ODP >>>> to adapt to widely differing internals among platforms. >>>> * Anyone can create an ODP implementation tailored to their platform >>>> -* Distribution and mainteinance of each implementation is as owner >>>> wishes >>>> +* Distribution and maintenance of each implementation is as owner >>>> wishes >>>> - Open source or closed source as business needs determine >>>> - Have independent release cycles and service streams >>>> * Allows HW and SW innovation in how ODP APIs are implemented on each >>>> platform. >>>> @@ -100,7 +103,7 @@ drivers supported by DPDK. >>>> they are derived from a reference implementation. >>>> >>>> .ODP Validation Test Suite >>>> -Third, to enure consistency between different ODP implementations, ODP >>>> +Third, to ensure consistency between different ODP implementations, ODP >>>> consists of a validation suite that verifies that any given >>>> implementation of >>>> ODP faithfully provides the specified functional behavior of each ODP >>>> API. >>>> As a separate open source component, the validation suite may be used >>>> by >>>> @@ -115,16 +118,16 @@ ODP API specification. >>>> * Key to ensuring application portability across all ODP >>>> implementations >>>> * Tests that ODP implementations conform to the specified functional >>>> behavior >>>> of ODP APIs. >>>> -* Can be run at any time by users and vendors to validat >>>> implementations >>>> -od ODP. >>>> +* Can be run at any time by users and vendors to validate >>>> implementations >>>> +of ODP. >>>> >>>> -=== ODP API Specification Versioning === >>>> +=== ODP API Specification Versioning >>>> As an evolving standard, the ODP API specification is released under an >>>> incrementing version number, and corresponding implementations of ODP, >>>> as well >>>> as the validation suite that verifies API conformance, are linked to >>>> this >>>> -version number. ODP versions are specified using a stanard three-level >>>> +version number. ODP versions are specified using a standard three-level >>>> number (major.minor.fixlevel) that are incremented according to the >>>> degree of >>>> -change the level represents. Increments to the fixlevel represent >>>> clarification >>>> +change the level represents. Increments to the fix level represent >>>> clarification >>>> of the specification or other minor changes that do not affect either >>>> the >>>> syntax or semantics of the specification. Such changes in the API >>>> specification >>>> are expected to be rare. Increments to the minor level >>>> @@ -136,26 +139,26 @@ the major level represent significant structural >>>> changes that most likely >>>> require some level of application source code change, again as >>>> documented in >>>> the release notes for that version. >>>> >>>> -=== ODP Implementation Versioning === >>>> +=== ODP Implementation Versioning >>>> ODP implementations are free to use whatever release naming/numbering >>>> conventions they wish, as long as it is clear what level of the ODP >>>> API a given >>>> release implements. A recommended convention is to use the same three >>>> level >>>> numbering scheme where the major and minor numbers correspond to the >>>> ODP API >>>> -level and the fixlevel represents an implementation-defined service >>>> level >>>> +level and the fix level represents an implementation-defined service >>>> level >>>> associated with that API level implementation. The LNG-supplied ODP >>>> reference >>>> implementations follow this convention. >>>> >>>> -=== ODP Validation Test Suite Versioning === >>>> +=== ODP Validation Test Suite Versioning >>>> The ODP validation test suite follows these same naming conventions. >>>> The major >>>> and minor release numbers correspond to the ODP API level that the >>>> suite >>>> -validates and the fixlevel represents the service level of the >>>> validation >>>> +validates and the fix level represents the service level of the >>>> validation >>>> suite itself for that API level. >>>> >>>> -=== ODP Design Goals === >>>> +=== ODP Design Goals >>>> ODP has three primary goals that follow from its component structure. >>>> The first >>>> is application portability across a wide range of platforms. These >>>> platforms >>>> differ in terms of processor instruction set architecture, number and >>>> types of >>>> -application processing cores, memory oranization, as well as the >>>> number and >>>> +application processing cores, memory organization, as well as the >>>> number and >>>> type of platform specific hardware acceleration and offload features >>>> that >>>> are available. ODP applications can move from one conforming >>>> implementation >>>> to another with at most a recompile. >>>> @@ -175,7 +178,7 @@ of processing cores that are available to realize >>>> application function. The >>>> result is that an application written to this model does not require >>>> redesign >>>> as it scales from 4, to 40, to 400 cores. >>>> >>>> -== Organization of this Document == >>>> +== Organization of this Document >>>> This document is organized into several sections. The first presents a >>>> high >>>> level overview of the ODP API component areas and their associated >>>> abstract >>>> data types. This section introduces ODP APIs at a conceptual level. >>>> @@ -190,14 +193,14 @@ full reference specification for each API. The >>>> latter is intended to be used >>>> by ODP application programmers, as well as implementers, to understand >>>> the >>>> precise syntax and semantics of each API. >>>> >>>> -== ODP API Concepts == >>>> +== ODP API Concepts >>>> ODP programs are built around several conceptual structures that every >>>> -appliation programmer needs to be familiar with to use ODP >>>> effectively. The >>>> +application programmer needs to be familiar with to use ODP >>>> effectively. The >>>> main ODP concepts are: >>>> Thread, Event, Queue, Pool, Shared Memory, Buffer, Packet, PktIO, >>>> Timer, >>>> and Synchronizer. >>>> >>>> -=== Thread === >>>> +=== Thread >>>> The thread is the fundamental programming unit in ODP. ODP >>>> applications are >>>> organized into a collection of threads that perform the work that the >>>> application is designed to do. ODP threads may or may not share memory >>>> with >>>> @@ -209,7 +212,7 @@ A control thread is a supervisory thread that >>>> organizes >>>> the operation of worker threads. Worker threads, by contrast, exist to >>>> perform the main processing logic of the application and employ a run >>>> to >>>> completion model. Worker threads, in particular, are intended to >>>> operate on >>>> -dedicated processing cores, especially in many core proessing >>>> environments, >>>> +dedicated processing cores, especially in many core processing >>>> environments, >>>> however a given implementation may multitask multiple threads on a >>>> single >>>> core if desired (typically on smaller and lower performance target >>>> environments). >>>> @@ -219,7 +222,7 @@ _thread mask_ and _scheduler group_ that determine >>>> where they can run and >>>> the type of work that they can handle. These will be discussed in >>>> greater >>>> detail later. >>>> >>>> -=== Event === >>>> +=== Event >>>> Events are what threads process to perform their work. Events can >>>> represent >>>> new work, such as the arrival of a packet that needs to be processed, >>>> or they >>>> can represent the completion of requests that have executed >>>> asynchronously. >>>> @@ -232,7 +235,7 @@ References to events are via handles of abstract >>>> type +odp_event_t+. Cast >>>> functions are provided to convert these into specific handles of the >>>> appropriate type represented by the event. >>>> >>>> -=== Queue === >>>> +=== Queue >>>> A queue is a message passing channel that holds events. Events can be >>>> added to a queue via enqueue operations or removed from a queue via >>>> dequeue >>>> operations. The endpoints of a queue will vary depending on how it is >>>> used. >>>> @@ -244,7 +247,7 @@ stateful processing on events as well as stateless >>>> processing. >>>> >>>> Queues are represented by handles of abstract type +odp_queue_t+. >>>> >>>> -=== Pool === >>>> +=== Pool >>>> A pool is a shared memory area from which elements may be drawn. Pools >>>> represent the backing store for events, among other things. Pools are >>>> typically created and destroyed by the application during >>>> initialization and >>>> @@ -256,32 +259,32 @@ are Buffer and Packet. >>>> >>>> Pools are represented by handles of abstract type +odp_pool_t+. >>>> >>>> -=== Shared Memory === >>>> +=== Shared Memory >>>> Shared memory represents raw blocks of storage that are sharable >>>> between >>>> threads. They are the building blocks of pools but can be used >>>> directly by >>>> ODP applications if desired. >>>> >>>> Shared memory is represented by handles of abstract type +odp_shm_t+. >>>> >>>> -=== Buffer === >>>> +=== Buffer >>>> A buffer is a fixed sized block of shared storage that is used by ODP >>>> components and/or applications to realize their function. Buffers >>>> contain >>>> zero or more bytes of application data as well as system maintained >>>> metadata that provide information about the buffer, such as its size >>>> or the >>>> pool it was allocated from. Metadata is an important ODP concept >>>> because it >>>> allows for arbitrary amounts of side information to be associated with >>>> an >>>> -ODP object. Most ODP objects have assocaited metadata and this >>>> metadata is >>>> +ODP object. Most ODP objects have associated metadata and this >>>> metadata is >>>> manipulated via accessor functions that act as getters and setters for >>>> -this information. Getter acces functions permit an application to read >>>> +this information. Getter access functions permit an application to read >>>> a metadata item, while setter access functions permit an application >>>> to write >>>> a metadata item. Note that some metadata is inherently read only and >>>> thus >>>> no setter is provided to manipulate it. When object have multiple >>>> metadata >>>> items, each has its own associated getter and/or setter access >>>> function to >>>> inspect or manipulate it. >>>> >>>> -Buffers are represened by handles of abstract type +odp_buffer_t+. >>>> +Buffers are represented by handles of abstract type +odp_buffer_t+. >>>> >>>> -=== Packet === >>>> +=== Packet >>>> Packets are received and transmitted via I/O interfaces and represent >>>> the basic data that data plane applications manipulate. >>>> Packets are drawn from pools of type +ODP_POOL_PACKET+. >>>> @@ -294,7 +297,7 @@ with each packet for its own use. >>>> >>>> Packets are represented by handles of abstract type +odp_packet_t+. >>>> >>>> -=== PktIO === >>>> +=== PktIO >>>> PktIO is how ODP represents I/O interfaces. A pktio object is a logical >>>> port capable of receiving and/or transmitting packets. This may be >>>> directly >>>> supported by the underlying platform as an integrated feature, >>>> @@ -302,7 +305,7 @@ or may represent a device attached via a PCIE or >>>> other bus. >>>> >>>> PktIOs are represented by handles of abstract type +odp_pktio_t+. >>>> >>>> -=== Timer === >>>> +=== Timer >>>> Timers are how ODP applications measure and respond to the passage of >>>> time. >>>> Timers are drawn from specialized pools called timer pools that have >>>> their >>>> own abstract type (+odp_timer_pool_t+). Applications may have many >>>> timers >>>> @@ -310,7 +313,7 @@ active at the same time and can set them to use >>>> either relative or absolute >>>> time. When timers expire they create events of type +odp_timeout_t+, >>>> which >>>> serve as notifications of timer expiration. >>>> >>>> -=== Synchronizer === >>>> +=== Synchronizer >>>> Multiple threads operating in parallel typically require various >>>> synchronization services to permit them to operate in a reliable and >>>> coordinated manner. ODP provides a rich set of locks, barriers, and >>>> similar >>>> @@ -325,7 +328,7 @@ flow of work through an ODP application. These >>>> include the Classifier, >>>> Scheduler, and Traffic Manager. These components relate to the three >>>> main stages of packet processing: Receive, Process, and Transmit. >>>> >>>> -=== Classifier === >>>> +=== Classifier >>>> The *Classifier* provides a suite of APIs that control packet receive >>>> (RX) >>>> processing. >>>> >>>> @@ -362,8 +365,8 @@ Note that the use of the classifier is optional. >>>> Applications may directly >>>> receive packets from a corresponding PktIO input queue via direct >>>> polling >>>> if they choose. >>>> >>>> -=== Scheduler === >>>> -The *Scheduler* provides a suite of APIs that control scalabable event >>>> +=== Scheduler >>>> +The *Scheduler* provides a suite of APIs that control scalable event >>>> processing. >>>> >>>> .ODP Scheduler and Event Processing >>>> @@ -391,10 +394,10 @@ scheduled back to a thread to continue processing >>>> with the results of the >>>> requested asynchronous operation. >>>> >>>> Threads themselves can enqueue events to queues for downstream >>>> processing >>>> -by other threads, permitting flexibility in how applicaitions structure >>>> +by other threads, permitting flexibility in how applications structure >>>> themselves to maximize concurrency. >>>> >>>> -=== Traffic Manager === >>>> +=== Traffic Manager >>>> The *Traffic Manager* provides a suite of APIs that control traffic >>>> shaping and >>>> Quality of Service (QoS) processing for packet output. >>>> >>>> @@ -413,23 +416,33 @@ goals. Again, the advantage here is that on many >>>> platforms traffic management >>>> functions are implemented in hardware, permitting transparent offload >>>> of >>>> this work. >>>> >>>> -Glossary >>>> --------- >>>> -[glossary] >>>> -odp_worker:: >>>> - An opd_worker is a type of odp_thread. It will usually be isolated >>>> from the scheduling of any host operating system and is intended for >>>> fast-path processing with a low and predictable latency. Odp_workers will >>>> not generally receive interrupts and will run to completion. >>>> -odp_control:: >>>> - An odp_control is a type of odp_thread. It will be isolated from >>>> the host operating system house keeping tasks but will be scheduled by it >>>> and may receive interrupts. >>>> -odp_thread:: >>>> - An odp_thread is a flow of execution that in a Linux environment >>>> could be a Linux process or thread. >>>> -event:: >>>> - An event is a notification that can be placed in a queue. >>>> - >>>> -The include structure >>>> ---------------------- >>>> -Applications only include the 'include/odp.h file which includes the >>>> 'platform/<implementation name>/include/plat' files to provide a complete >>>> definition of the API on that platform. >>>> -The doxygen documentation defining the behavior of the ODP API is all >>>> contained in the public API files, and the actual definitions for an >>>> implementation will be found in the per platform directories. >>>> -Per-platform data that might normally be a #define can be recovered >>>> via the appropriate access function if the #define is not directly visible >>>> to the application. >>>> +== ODP Application Programming >>>> +At the highest level, an *ODP Application* is a program that uses one >>>> or more >>>> +ODP APIs. Because ODP is a framework rather than a programming >>>> environment, >>>> +applications are free to also use other APIs that may or may not >>>> provide the >>>> +same portability characteristics as ODP APIs. >>>> + >>>> +ODP applications vary in terms of what they do and how they operate, >>>> but in >>>> +general all share the following characteristics: >>>> + >>>> +. They are organized into one or more _threads_ that execute in >>>> parallel. >>>> +. These threads communicate and coordinate their activities using >>>> various >>>> +_synchronization_ mechanisms. >>>> +. They receive packets from one or more _packet I/O interfaces_. >>>> +. They examine, transform, or otherwise process packets. >>>> +. They transmit packets to one or more _packet I/O interfaces_. >>>> + >>>> +ODP provides APIs to assist in each of these areas. >>>> + >>>> +=== The include structure >>>> +Applications only include the 'include/odp.h' file, which includes the >>>> +'platform/<implementation name>/include/odp' files to provide a >>>> complete >>>> +definition of the API on that platform. The doxygen documentation >>>> defining >>>> +the behavior of the ODP API is all contained in the public API files, >>>> and the >>>> +actual definitions for an implementation will be found in the per >>>> platform >>>> +directories. Per-platform data that might normally be a +#define+ can >>>> be >>>> +recovered via the appropriate access function if the #define is not >>>> directly >>>> +visible to the application. >>>> >>>> .Users include structure >>>> ---- >>>> @@ -442,51 +455,305 @@ Per-platform data that might normally be a >>>> #define can be recovered via the appr >>>> │ └── odp.h This file should be the only file included by the >>>> application. >>>> ---- >>>> >>>> -Initialization >>>> --------------- >>>> -IMPORTANT: ODP depends on the application to perform a graceful >>>> shutdown, calling the terminate functions should only be done when the >>>> application is sure it has closed the ingress and subsequently drained all >>>> queues etc. >>>> +=== Initialization >>>> +IMPORTANT: ODP depends on the application to perform a graceful >>>> shutdown, >>>> +calling the terminate functions should only be done when the >>>> application is >>>> +sure it has closed the ingress and subsequently drained all queues, >>>> etc. >>>> + >>>> +=== Startup >>>> +The first API that must be called by an ODP application is >>>> 'odp_init_global()'. >>>> +This takes two pointers. The first, +odp_init_t+, contains ODP >>>> initialization >>>> +data that is platform independent and portable, while the second, >>>> ++odp_platform_init_t+, is passed unparsed to the implementation >>>> +to be used for platform specific data that is not yet, or may never be >>>> +suitable for the ODP API. >>>> + >>>> +Calling odp_init_global() establishes the ODP API framework and MUST be >>>> +called before any other ODP API may be called. Note that it is only >>>> called >>>> +once per application. Following global initialization, each thread in >>>> turn >>>> +calls 'odp_init_local()' is called. This establishes the local ODP >>>> thread >>>> +context for that thread and MUST be called before other ODP APIs may be >>>> +called by that thread. >>>> + >>>> +=== Shutdown >>>> +Shutdown is the logical reverse of the initialization procedure, with >>>> +'odp_term_local()' called for each thread before 'odp_term_global()' is >>>> +called to terminate ODP. >>>> + >>>> +.ODP Application Structure Flow Diagram >>>> +image::../images/resource_management.png[align="center"] >>>> >>>> -Startup >>>> -~~~~~~~~ >>>> -The first API that must be called is 'odp_init_global()'. >>>> -This takes two pointers, odp_init_t contains ODP initialization data >>>> that is platform independent and portable. >>>> -The second odp_platform_init_t is passed un parsed to the >>>> implementation and can be used for platform specific data that is not yet, >>>> or may never be suitable for the ODP API. >>>> +== Common Conventions >>>> +Many ODP APIs share common conventions regarding their arguments and >>>> return >>>> +types. This section highlights some of the more common and frequently >>>> used >>>> +conventions. >>>> + >>>> +=== Handles and Special Designators >>>> +ODP resources are represented via _handles_ that have abstract type >>>> +_odp_resource_t_. So pools are represented by handles of type >>>> +odp_pool_t+, >>>> +queues by handles of type +odp_queue_t+, etc. Each such type >>>> +has a distinguished type _ODP_RESOURCE_INVALID_ that is used to >>>> indicate a >>>> +handle that does not refer to a valid resource of that type. Resources >>>> are >>>> +typically created via an API named _odp_resource_create()_ that >>>> returns a >>>> +handle of type _odp_resource_t_ that represents the created object. >>>> This >>>> +returned handle is set to _ODP_RESOURCE_INVALID_ if, for example, the >>>> +resource could not be created due to resource exhaustion. Invalid >>>> resources >>>> +do not necessarily represent error conditions. For example, >>>> +ODP_EVENT_INVALID+ >>>> +in response to an +odp_queue_deq()+ call to get an event from a queue >>>> simply >>>> +indicates that the queue is empty. >>>> + >>>> +=== Addressing Scope >>>> +Unless specifically noted in the API, all ODP resources are global to >>>> the ODP >>>> +application, whether it runs as a single process or multiple >>>> processes. ODP >>>> +handles therefore have common meaning within an ODP application but >>>> have no >>>> +meaning outside the scope of the application. >>>> + >>>> +=== Resources and Names >>>> +Many ODP resource objects, such as pools and queues, support an >>>> +application-specified character string _name_ that is associated with >>>> an ODP >>>> +object at create time. This name serves two purposes: documentation, >>>> and >>>> +lookup. The lookup function is particularly useful to allow an ODP >>>> application >>>> +that is divided into multiple processes to obtain the handle for the >>>> common >>>> +resource. >>>> + >>>> +== Queues >>>> +Queues are the fundamental event sequencing mechanism provided by ODP >>>> and all >>>> +ODP applications make use of them either explicitly or implicitly. >>>> Queues are >>>> +created via the 'odp_queue_create()' API that returns a handle of type >>>> ++odp_queue_t+ that is used to refer to this queue in all subsequent >>>> APIs that >>>> +reference it. Queues have one of two ODP-defined _types_, POLL, and >>>> SCHED that >>>> +determine how they are used. POLL queues directly managed by the ODP >>>> +application while SCHED queues make use of the *ODP scheduler* to >>>> provide >>>> +automatic scalable dispatching and synchronization services. >>>> + >>>> +.Operations on POLL queues >>>> +[source,c] >>>> +---- >>>> +odp_queue_t poll_q1 = odp_queue_create("poll queue 1", >>>> ODP_QUEUE_TYPE_POLL, NULL); >>>> +odp_queue_t poll_q2 = odp_queue_create("poll queue 2", >>>> ODP_QUEUE_TYPE_POLL, NULL); >>>> +... >>>> +odp_event_t ev = odp_queue_deq(poll_q1); >>>> +...do something >>>> +int rc = odp_queue_enq(poll_q2, ev); >>>> +---- >>>> >>>> -The second API that must be called is 'odp_init_local()', this must be >>>> called once per odp_thread, regardless of odp_thread type. Odp_threads may >>>> be of type ODP_THREAD_WORKER or ODP_THREAD_CONTROL >>>> +The key distinction is that dequeueing events from POLL queues is an >>>> +application responsibility while dequeueing events from SCHED queues >>>> is the >>>> +responsibility of the ODP scheduler. >>>> >>>> -Shutdown >>>> -~~~~~~~~~ >>>> -Shutdown is the logical reverse of the initialization procedure, with >>>> 'odp_thread_term()' called for each worker before 'odp_term_global()' is >>>> called. >>>> +.Operations on SCHED queues >>>> +[source,c] >>>> +---- >>>> +odp_queue_param_t qp; >>>> +odp_queue_param_init(&qp); >>>> +odp_schedule_prio_t prio = ...; >>>> +odp_schedule_group_t sched_group = ...; >>>> +qp.sched.prio = prio; >>>> +qp.sched.sync = ODP_SCHED_SYNC_[NONE|ATOMIC|ORDERED]; >>>> +qp.sched.group = sched_group; >>>> +qp.lock_count = n; /* Only relevant for ordered queues */ >>>> +odp_queue_t sched_q1 = odp_queue_create("sched queue 1", >>>> ODP_QUEUE_TYPE_SCHED, &qp); >>>> + >>>> +...thread init processing >>>> + >>>> +while (1) { >>>> + odp_event_t ev; >>>> + odp_queue_t which_q; >>>> + ev = odp_schedule(&which_q, <wait option>); >>>> + ...process the event >>>> +} >>>> +---- >>>> >>>> -image::../images/resource_management.png[align="center"] >>>> +With scheduled queues, events are sent to a queue, and the the sender >>>> chooses >>>> +a queue based on the service it needs. The sender does not need to know >>>> +which ODP thread (on which core) or hardware accelerator will process >>>> +the event, but all the events on a queue are eventually scheduled and >>>> processed. >>>> + >>>> +As can be seen, SCHED queues have additional attributes that are >>>> specified at >>>> +queue create that control how the scheduler is to process events >>>> contained >>>> +on them. These include group, priority, and synchronization class. >>>> + >>>> +=== Scheduler Groups >>>> +The scheduler's dispatching job is to return the next event from the >>>> highest >>>> +priority SCHED queue that the caller is eligible to receive events >>>> from. >>>> +This latter consideration is determined by the queues _scheduler >>>> group_, which >>>> +is set at queue create time, and by the caller's _scheduler group >>>> mask_ that >>>> +indicates which scheduler group(s) it belongs to. Scheduler groups are >>>> +represented by handles of type +odp_scheduler_group_t+ and are created >>>> by >>>> +the *odp_scheduler_group_create()* API. A number of scheduler groups >>>> are >>>> +_predefined_ by ODP. These include +ODP_SCHED_GROUP_ALL+ (all >>>> threads), >>>> ++ODP_SCHED_GROUP_WORKER+ (all worker threads), and >>>> +ODP_SCHED_GROUP_CONTROL+ >>>> +(all control threads). The application is free to create additional >>>> scheduler >>>> +groups for its own purpose and threads can join or leave scheduler >>>> groups >>>> +using the *odp_scheduler_group_join()* and >>>> *odp_scheduler_group_leave()* APIs >>>> + >>>> +=== Scheduler Priority >>>> +The +prio+ field of the +odp_queue_param_t+ specifies the queue's >>>> scheduling >>>> +priority, which is how queues within eligible scheduler groups are >>>> selected >>>> +for dispatch. Queues have a default scheduling priority of NORMAL but >>>> can be >>>> +set to HIGHEST or LOWEST according to application needs. >>>> + >>>> +=== Scheduler Synchronization >>>> +In addition to its dispatching function, which provide automatic >>>> scalability to >>>> +ODP applications in many core environments, the other main function of >>>> the >>>> +scheduler is to provide event synchronization services that greatly >>>> simplify >>>> +application programming in a parallel processing environment. A queue's >>>> +SYNC mode determines how the scheduler handles the synchronization >>>> processing >>>> +of multiple events originating from the same queue. >>>> + >>>> +Three types of queue scheduler synchronization area supported: >>>> Parallel, >>>> +Atomic, and Ordered. >>>> + >>>> +==== Parallel Queues >>>> +SCHED queues that specify a sync mode of ODP_SCHED_SYNC_NONE are >>>> unrestricted >>>> +in how events are processed. >>>> + >>>> +.Parallel Queue Scheduling >>>> +image::../images/parallel_queue.png[align="center"] >>>> >>>> -Queues >>>> ------- >>>> -There are three queue types, atomic, ordered and parallel. >>>> -A queue belongs to a single odp_worker and a odp_worker may have >>>> multiple queues. >>>> +All events held on parallel queues are eligible to be scheduled >>>> simultaneously >>>> +and any required synchronization between them is the responsibility of >>>> the >>>> +application. Events originating from parallel queues thus have the >>>> highest >>>> +throughput rate, however they also potentially involve the most work >>>> on the >>>> +part of the application. In the Figure above, four threads are calling >>>> +*odp_schedule()* to obtain events to process. The scheduler has >>>> assigned >>>> +three events from the first queue to three threads in parallel. The >>>> fourth >>>> +thread is processing a single event from the third queue. The second >>>> queue >>>> +might either be empty, of lower priority, or not in a scheduler group >>>> matching >>>> +any of the threads being serviced by the scheduler. >>>> + >>>> +=== Atomic Queues >>>> +Atomic queues simplify event synchronization because only a single >>>> event >>>> +from a given atomic queue may be processed at a time. Events scheduled >>>> from >>>> +atomic queues thus can be processed lock free because the locking is >>>> being >>>> +done implicitly by the scheduler. >>>> + >>>> +.Atomic Queue Scheduling >>>> +image::../images/atomic_queue.png[align="center"] >>>> >>>> -Events are sent to a queue, and the the sender chooses a queue based >>>> on the service it needs. >>>> -The sender does not need to know which odp_worker (on which core) or >>>> HW accelerator will process the event, but all the events on a queue are >>>> eventually scheduled and processed. >>>> +In this example, no matter how many events may be held in an atomic >>>> queue, only >>>> +one of them can be scheduled at a time. Here two threads process >>>> events from >>>> +two different atomic queues. Note that there is no synchronization >>>> between >>>> +different atomic queues, only between events originating from the same >>>> atomic >>>> +queue. The queue context associated with the atomic queue is held >>>> until the >>>> +next call to the scheduler or until the application explicitly >>>> releases it >>>> +via a call to *odp_schedule_release_atomic()*. >>>> >>>> -NOTE: Both ordered and parallel queue types improve throughput over an >>>> atomic queue (due to parallel event processing), but the user has to take >>>> care of the context data synchronization (if needed). >>>> +Note that while atomic queues simplify programming, the serial nature >>>> of >>>> +atomic queues will impair scaling. >>>> >>>> -Atomic Queue >>>> -~~~~~~~~~~~~ >>>> -Only one event at a time may be processed from a given queue. The >>>> processing maintains order and context data synchronization but this will >>>> impair scaling. >>>> +=== Ordered Queues >>>> +Ordered queues provide the best of both worlds by providing the >>>> inherent >>>> +scaleabilty of parallel queues, with the easy synchronization of atomic >>>> +queues. >>>> >>>> -.Overview Atomic Queue processing >>>> -image::../images/atomic_queue.png[align="center"] >>>> +.Ordered Queue Scheduling >>>> +image::../images/ordered_queue.png[align="center"] >>>> >>>> -Ordered Queue >>>> -~~~~~~~~~~~~~ >>>> -An ordered queue will ensure that the sequencing at the output is >>>> identical to that of the input, but multiple events may be processed >>>> simultaneously and the order is restored before the events move to the next >>>> queue >>>> +When scheduling events from an ordered queue, the scheduler dispatches >>>> multiple >>>> +events from the queue in parallel to different threads, however the >>>> scheduler >>>> +also ensures that the relative sequence of these events on output >>>> queues >>>> +is identical to their sequence from their originating ordered queue. >>>> + >>>> +As with atomic queues, the ordering guarantees associated with ordered >>>> queues >>>> +refer to events originating from the same queue, not for those >>>> originating on >>>> +different queues. Thus in this figure three thread are processing >>>> events 5, 3, >>>> +and 4, respectively from the first ordered queue. Regardless of how >>>> these >>>> +threads complete processing, these events will appear in their original >>>> +relative order on their output queue. >>>> + >>>> +==== Order Preservation >>>> +Relative order is preserved independent of whether events are being >>>> sent to >>>> +different output queues. For example, if some events are sent to >>>> output queue >>>> +A while others are sent to output queue B then the events on these >>>> output >>>> +queues will still be in the same relative order as they were on their >>>> +originating queue. Similarly, if the processing consumes events so >>>> that no >>>> +output is issued for some of them (_e.g.,_ as part of IP fragment >>>> reassembly >>>> +processing) then other events will still be correctly ordered with >>>> respect to >>>> +these sequence gaps. Finally, if multiple events are enqueued for a >>>> given >>>> +order (_e.g.,_ as part of packet segmentation processing for MTU >>>> +considerations), then each of these events will occupy the originator's >>>> +sequence in the target output queue(s). In this case the relative >>>> order of these >>>> +events will be in the order that the thread issued *odp_queue_enq()* >>>> calls for >>>> +them. >>>> + >>>> +The ordered context associated with the dispatch of an event from an >>>> ordered >>>> +queue lasts until the next scheduler call or until explicitly released >>>> by >>>> +the thread calling *odp_schedule_release_ordered()*. This call may be >>>> used >>>> +as a performance advisory that the thread no longer requires ordering >>>> +guarantees for the current context. As a result, any subsequent >>>> enqueues >>>> +within the current scheduler context will be treated as if the thread >>>> was >>>> +operating in a parallel queue context. >>>> + >>>> +==== Ordered Locking >>>> +Another powerful feature of the scheduler's handling of ordered queues >>>> is >>>> +*ordered locks*. Each ordered queue has associated with it a number of >>>> ordered >>>> +locks as specified by the _lock_count_ parameter at queue create time. >>>> + >>>> +Ordered locks provide an efficient means to perform in-order sequential >>>> +processing within an ordered context. For example, supposed events >>>> with relative >>>> +order 5, 6, and 7 are executing in parallel by three different >>>> threads. An >>>> +ordered lock will enable these threads to synchronize such that they >>>> can >>>> +perform some critical section in their originating queue order. The >>>> number of >>>> +ordered locks supported for each ordered queue is implementation >>>> dependent (and >>>> +queryable via the *odp_config_max_ordered_locks_per_queue()* API). If >>>> the >>>> +implementation supports multiple ordered locks then these may be used >>>> to >>>> +protect different ordered critical sections within a given ordered >>>> context. >>>> + >>>> +==== Summary: Ordered Queues >>>> +To see how these considerations fit together, consider the following >>>> code: >>>> + >>>> +.Processing with Ordered Queues >>>> +[source,c] >>>> +---- >>>> +void worker_thread() >>>> +{ >>>> + odp_init_local(); >>>> + ...other initialization processing >>>> + >>>> + while (1) { >>>> + ev = odp_schedule(&which_q, ODP_SCHED_WAIT); >>>> + ...process events in parallel >>>> + odp_schedule_order_lock(0); >>>> + ...critical section processed in order >>>> + odp_schedule_order_unlock(0); >>>> + ...continue processing in parallel >>>> + odp_queue_enq(dest_q, ev); >>>> + } >>>> +} >>>> +---- >>>> >>>> -.Overview Ordered Queue processing >>>> -image::../images/ordered_queue.png[align="center"] >>>> +This represents a simplified structure for a typical worker thread >>>> operating >>>> +on ordered queues. Multiple events are processed in parallel and the >>>> use of >>>> +ordered queues ensures that they will be placed on +dest_q+ in the >>>> same order >>>> +as they originated. While processing in parallel, the use of ordered >>>> locks >>>> +enables critical sections to be processed in order within the overall >>>> parallel >>>> +flow. When a thread arrives at the _odp_schedule_order_lock()_ call, >>>> it waits >>>> +until the locking order for this lock for all prior events has been >>>> resolved >>>> +and then enters the critical section. The >>>> _odp_schedule_order_unlock()_ call >>>> +releases the critical section and allows the next order to enter it. >>>> >>>> -Parallel Queue >>>> -~~~~~~~~~~~~~~ >>>> -There are no restrictions on the number of events being processed. >>>> +=== Queue Scheduling Summary >>>> >>>> -.Overview parallel Queue processing >>>> -image::../images/parallel_queue.png[align="center"] >>>> +NOTE: Both ordered and parallel queues improve throughput over atomic >>>> queues >>>> +due to parallel event processing, but require that the application take >>>> +steps to ensure context data synchronization if needed. >>>> + >>>> +== Glossary >>>> +[glossary] >>>> +worker thread:: >>>> + A worker is a type of ODP thread. It will usually be isolated from >>>> + the scheduling of any host operating system and is intended for >>>> fast-path >>>> + processing with a low and predictable latency. Worker threads will >>>> not >>>> + generally receive interrupts and will run to completion. >>>> +control thread:: >>>> + A control threadis a type of ODP thread. It will be isolated from >>>> the host >>>> + operating system house keeping tasks but will be scheduled by it >>>> and may >>>> + receive interrupts. >>>> +thread:: >>>> + An ODP thread is a flow of execution that in a Linux environment >>>> could be >>>> + a Linux process or thread. >>>> +event:: >>>> + An event is a notification that can be placed in a queue. >>>> +queue:: >>>> + A communication channel that holds events >>>> -- >>>> 2.1.4 >>>> >>>> _______________________________________________ >>>> lng-odp mailing list >>>> lng-odp@lists.linaro.org >>>> https://lists.linaro.org/mailman/listinfo/lng-odp >>>> >>> >>> >>> >>> -- >>> Mike Holmes >>> Technical Manager - Linaro Networking Group >>> Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM >>> SoCs >>> >>> >>> >> > > > -- > Mike Holmes > Technical Manager - Linaro Networking Group > Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs > > > -- Mike Holmes Technical Manager - Linaro Networking Group Linaro.org <http://www.linaro.org/> *│ *Open source software for ARM SoCs
_______________________________________________ lng-odp mailing list lng-odp@lists.linaro.org https://lists.linaro.org/mailman/listinfo/lng-odp
