Good points Brian. I can add some data point with real life observations of a DPI box analyzing two 10Gbps links (four 10 Gbps ports): there are approximately 50M known flows (TCP, UDP...), may be not active (TCP Close_wait).
FF On 10 November 2016 at 08:56, Brian Brooks <[email protected]> wrote: > On 11/07 16:46:12, Bala Manoharan wrote: > > Hi, > > Hiya > > > This mail thread discusses the design of classification queue group > > RFC. The same can be found in the google doc whose link is given > > below. > > Users can provide their comments either in this mail thread or in the > > google doc as per their convenience. > > > > https://docs.google.com/document/d/1fOoG9WDR0lMpVjgMAsx8QsMr0YFK9 > slR93LZ8VXqM2o/edit?usp=sharing > > > > The basic issues with queues as being a single target for a CoS are two > fold: > > > > Queues must be created and deleted individually. This imposes a > > significant burden when queues are used to represent individual flows > > since the application may need to process thousands (or millions) of > > flows. > > Wondering why there is an issue with creating and deleting queues > individually > if queue objects represent millions of flows.. > > Could an application ever call odp_schedule() and receive an event (e.g. > packet) > from a queue (of opaque type odp_queue_t) and that queue has never been > created > by the application (via odp_queue_create())? Could that ever happen from > the > hardware, and could the application ever handle that? > > Or, is it related to memory usage? The reference implementation > struct queue_entry_s is 320 bytes on a 64-bit machine. > > 2^28 ~= 268,435,456 queues -> 81.920 GB > 2^26 ~= 67,108,864 queues -> 20.480 GB > 2^22 ~= 4,194,304 queues -> 1.280 GB > > Forget about 320 bytes per queue, if each queue was represented by a 32-bit > integer (4 bytes!) the usage would be: > > 2^28 ~= 268,435,456 queues -> 1.024 GB > 2^26 ~= 67,108,864 queues -> 256 MB > 2^22 ~= 4,194,304 queues -> 16 MB > > That still might be a lot of usage if the application must explicitly > create > every queue (before it is used) and require an ODP implementation to map > between every ODP queue object (opaque type) and the internal queue. > > Lets say ODP API has two classes of handles: 1) pointers, 2) integers. An > opaque > pointer is used to point to some other software object. This object should > be > larger than 64 bits (or 32 bits on a chip in 32-bit pointer mode) > otherwise it > could just be represented in a 64-bit (or 32-bit) integer type value! > > To support millions of queues (flows) should odp_queue_t be an integer > type in > the API? A software-only implementation may still use 320 bytes per queue > and > use that integer as an index into an array or as a key for lookup > operation on a > data structure containing queues. An implementation with hardware assist > may > use this integer value directly when interfacing with hardware! > > Would it still be necessary to assign a "name" to each queue (flow)? > > Would a queue (flow) also require an "op type" to explicitly specify > whether > access to the queue (flow) is threadsafe? Atomic queues are threadsafe > since > only 1 core at any given time can recieve from it. Parallel queues are also > threadsafe. Are all ODP APIs threadsafe? > > > A single PMR can only match a packet to a single queue associated with > > a target CoS. This prohibits efficient capture of subfield > > classification. > > odp_cls_pmr_create() can take more than 1 odp_pmr_param_t, so it is > possible > to create a single PMR which matches multiple fields of a packet. I can > imagine > a case where a packet matches pmr1 (match Vlan) and also matches pmr2 > (match Vlan AND match L3DestIP). Is that an example of subfield > classification? > How does the queue relate? > > > To solve these issues, Tiger Moth introduces the concept of a queue > > group. A queue group is an extension to the existing queue > > specification in a Class of Service. > > > > Queue groups solve the classification issues associated with > > individual queues in three ways: > > > > * The odp_queue_group_create() API can create a large number of > > related queues with a single call. > > If the application calls this API, does that mean the ODP implementation > can create a large number of queues? What happens if the application > receives an event on a queue that was created by the implmentation--how > does the application know whether this queue was created by the hardware > according to the ODP Classification or whether the queue was created by > the application? > > > * A single PMR can spread traffic to many queues associated with the > > same CoS by assigning packets matching the PMR to a queue group rather > > than a queue. > > * A hashed PMR subfield is used to distribute individual queues within > > a queue group for scheduling purposes. > > Is there a way to write a test case for this? Trying to think of what kind > of > packets (traffic distribution) and how those packets would get classified > and > get assigned to queues. > > > diff --git a/include/odp/api/spec/classification.h > > b/include/odp/api/spec/classification.h > > index 6eca9ab..cf56852 100644 > > --- a/include/odp/api/spec/classification.h > > +++ b/include/odp/api/spec/classification.h > > @@ -126,6 +126,12 @@ typedef struct odp_cls_capability_t { > > > > /** A Boolean to denote support of PMR range */ > > odp_bool_t pmr_range_supported; > > + > > + /** A Boolean to denote support of queue group */ > > + odp_bool_t queue_group_supported; > > + > > + /** A Boolean to denote support of queue */ > > + odp_bool_t queue_supported; > > } odp_cls_capability_t; > > > > > > /** > > @@ -162,7 +168,18 @@ typedef enum { > > * Used to communicate class of service creation options > > */ > > typedef struct odp_cls_cos_param { > > - odp_queue_t queue; /**< Queue associated with CoS */ > > + /** If type is ODP_QUEUE_T, odp_queue_t is linked with CoS, > > + * if type is ODP_QUEUE_GROUP_T, odp_queue_group_t is linked with CoS. > > + */ > > + odp_queue_type_e type; > > + > > + typedef union { > > + /** Queue associated with CoS */ > > + odp_queue_t queue; > > + > > + /** Queue Group associated with CoS */ > > + odp_queue_group_t queue_group; > > + }; > > odp_pool_t pool; /**< Pool associated with CoS */ > > odp_cls_drop_t drop_policy; /**< Drop policy associated with CoS */ > > } odp_cls_cos_param_t; > > > > > > diff --git a/include/odp/api/spec/queue.h b/include/odp/api/spec/queue.h > > index 51d94a2..7dde060 100644 > > --- a/include/odp/api/spec/queue.h > > +++ b/include/odp/api/spec/queue.h > > @@ -158,6 +158,87 @@ typedef struct odp_queue_param_t { > > odp_queue_t odp_queue_create(const char *name, const odp_queue_param_t > *param); > > > > +/** > > + * Queue group capability > > + * This capability structure defines system Queue Group capability > > + */ > > +typedef struct odp_queue_group_capability_t { > > + /** Number of queues supported per queue group */ > > + unsigned supported_queues; > > + /** Supported protocol fields for hashing*/ > > + odp_pktin_hash_proto_t supported; > > +} > > + > > +/** > > + * ODP Queue Group parameters > > + * Queue group supports only schedule queues <TBD??> > > + */ > > +typedef struct odp_queue_group_param_t { > > + /** Number of queue to be created for this queue group > > + * implementation may round up the value to nearest power of 2 > > + * and value should be less than the number of queues > > + * supported per queue group > > + */ > > + unsigned num_queue; > > + > > + /** Protocol field selection for queue group distribution > > + * Multiple fields can be selected in combination > > + */ > > + odp_queue_group_hash_proto_t hash; > > + > > +} odp_queue_group_param_t; > > + > > +/** > > + * Initialize queue group params > > + * > > + * Initialize an odp_queue_group_param_t to its default values for all > fields. > > + * > > + * @param param Address of the odp_queue_group_param_t to be > initialized > > + */ > > +void odp_queue_group_param_init(odp_queue_group_param_t *param); > > + > > +/** > > + * Queue Group create > > + * > > + * Create a queue group according to the queue group parameters. > > + * The individual queues belonging to a queue group are created by the > > + * implementation and the distribution of packets into those queues are > > + * decided based on the odp_queue_group_hash_proto_t parameters. > > + * The individual queues within a queue group are both created and > deleted > > + * by the implementation. > > + * > > + * @param name Queue Group name > > + * @param param Queue Group parameters. > > + * > > + * @return Queue group handle > > + * @retval ODP_QUEUE_GROUP_INVALID on failure > > + */ > > +odp_queue_group_t odp_queue_group_create(const char *name, > > + const odp_queue_group_param_t *param); > > Regards, > > Bala > -- [image: Linaro] <http://www.linaro.org/> François-Frédéric Ozog | *Director Linaro Networking Group* T: +33.67221.6485 [email protected] | Skype: ffozog
