Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
Alll we want to get this change in to help define the 1.0 API, can we get reviews finalised for it. On 5 November 2014 03:00, Alexandru Badicioiu < alexandru.badici...@linaro.org> wrote: > The specific case I was talking about is when a packet is dequeued from an > ordered queue and it needs fragmentation as part of the processing before > the next enqueue which is supposed to restore the order - for example IPsec > red side fragmentation (fragmentation before encryption) or fragmentation > before transmission on an interface having a MTU smaller than the packet > size. > The fragments of the same packet are ordered with respect to each other > but they can be interleaved with other packets from the same queue or with > fragments of another packet. > As we have skip order function which essentially tells the order > restoration logic to give up on the current expected packet in sequence and > move to the next packet we also need a way to tell that a sequence of > buffers share the same ordering information so the order restoration logic > treats them as a whole rather as separate packets. > > Alex > > > > > On 4 November 2014 17:15, Gilad Ben Yossef wrote: > >> >> >> I don't think it can use the first fragment place in the order. It can >> use the last fragment to arrive place in the queue. >> >> Otherwise you have to stop forwarding until all fragments arrive to keep >> the order… no? >> >> >> >> *Gilad Ben-Yossef* >> >> Software Architect >> >> EZchip Technologies Ltd. >> >> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel >> >> Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 >> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 >> >> Email: gil...@ezchip.com, Web: http://www.ezchip.com >> >> >> >> *"Ethernet always wins."* >> >> — Andy Bechtolsheim >> >> >> >> *From:* Alexandru Badicioiu [mailto:alexandru.badici...@linaro.org] >> *Sent:* Tuesday, November 04, 2014 4:13 PM >> *To:* Bill Fischofer >> *Cc:* Gilad Ben Yossef; Savolainen, Petri (NSN - FI/Espoo); >> lng-odp@lists.linaro.org >> >> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions >> >> >> >> In addition to skip order function I think we need to handle the case >> when a packet gets fragmented (e.g. IP fragmentation). In this case the >> fragments should share the original sequence number and the order >> restoration logic should be aware when the last fragment occurs. >> >> >> >> Alex >> >> >> >> On 3 November 2014 13:45, Bill Fischofer >> wrote: >> >> OK, thanks. That certainly matches the step-by-step semantics Petri >> proposed. We'll have to discuss what, if anything, more we need regarding >> ORDERED queues post-v1.0 next year. >> >> >> >> Bill >> >> >> >> On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef >> wrote: >> >> >> >> >> >> What NPS does in HW is that each buffer you dequeue actually stays on the >> queue (but is marked as consumed by the application) and you are given a >> "tag" for it. >> >> >> >> When I want to queue the buffer to the next, I give the HW the tag I got, >> not the buffer handle and what the HW does is only than fully dequeue the >> buffer from the original queue before queuing it into the new queue – but >> only after all preceding (older) buffers on the original queue have been >> fully dequeued, >> >> >> >> Free of a buffer causes a dequeue from the original queue as well. >> >> >> >> And I can also give release a buffer from the original queue (give the HW >> the tag back and say "you don't have to wait for this buffer any longer). >> >> >> >> I hope this helps, >> >> Gilad >> >> >> >> *Gilad Ben-Yossef* >> >> Software Architect >> >> EZchip Technologies Ltd. >> >> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel >> >> Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 >> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 >> >> Email: gil...@ezchip.com, Web: http://www.ezchip.com >> >> >> >> *"Ethernet always wins."* >> >> — Andy Bechtolsheim >> >> >> >> *From:* Bill Fischofer [mailto:bill.fischo...@linaro.org] >> *Sent:* Sunday, November 02, 2014 5:44 PM >> *To:* Gilad Ben Yossef >> *Cc:* Savola
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
The specific case I was talking about is when a packet is dequeued from an ordered queue and it needs fragmentation as part of the processing before the next enqueue which is supposed to restore the order - for example IPsec red side fragmentation (fragmentation before encryption) or fragmentation before transmission on an interface having a MTU smaller than the packet size. The fragments of the same packet are ordered with respect to each other but they can be interleaved with other packets from the same queue or with fragments of another packet. As we have skip order function which essentially tells the order restoration logic to give up on the current expected packet in sequence and move to the next packet we also need a way to tell that a sequence of buffers share the same ordering information so the order restoration logic treats them as a whole rather as separate packets. Alex On 4 November 2014 17:15, Gilad Ben Yossef wrote: > > > I don't think it can use the first fragment place in the order. It can use > the last fragment to arrive place in the queue. > > Otherwise you have to stop forwarding until all fragments arrive to keep > the order… no? > > > > *Gilad Ben-Yossef* > > Software Architect > > EZchip Technologies Ltd. > > 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel > > Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 > Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 > > Email: gil...@ezchip.com, Web: http://www.ezchip.com > > > > *"Ethernet always wins."* > > — Andy Bechtolsheim > > > > *From:* Alexandru Badicioiu [mailto:alexandru.badici...@linaro.org] > *Sent:* Tuesday, November 04, 2014 4:13 PM > *To:* Bill Fischofer > *Cc:* Gilad Ben Yossef; Savolainen, Petri (NSN - FI/Espoo); > lng-odp@lists.linaro.org > > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > In addition to skip order function I think we need to handle the case when > a packet gets fragmented (e.g. IP fragmentation). In this case the > fragments should share the original sequence number and the order > restoration logic should be aware when the last fragment occurs. > > > > Alex > > > > On 3 November 2014 13:45, Bill Fischofer > wrote: > > OK, thanks. That certainly matches the step-by-step semantics Petri > proposed. We'll have to discuss what, if anything, more we need regarding > ORDERED queues post-v1.0 next year. > > > > Bill > > > > On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef > wrote: > > > > > > What NPS does in HW is that each buffer you dequeue actually stays on the > queue (but is marked as consumed by the application) and you are given a > "tag" for it. > > > > When I want to queue the buffer to the next, I give the HW the tag I got, > not the buffer handle and what the HW does is only than fully dequeue the > buffer from the original queue before queuing it into the new queue – but > only after all preceding (older) buffers on the original queue have been > fully dequeued, > > > > Free of a buffer causes a dequeue from the original queue as well. > > > > And I can also give release a buffer from the original queue (give the HW > the tag back and say "you don't have to wait for this buffer any longer). > > > > I hope this helps, > > Gilad > > > > *Gilad Ben-Yossef* > > Software Architect > > EZchip Technologies Ltd. > > 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel > > Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 > Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 > > Email: gil...@ezchip.com, Web: http://www.ezchip.com > > > > *"Ethernet always wins."* > > — Andy Bechtolsheim > > > > *From:* Bill Fischofer [mailto:bill.fischo...@linaro.org] > *Sent:* Sunday, November 02, 2014 5:44 PM > *To:* Gilad Ben Yossef > *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org > > > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > Thanks Gilad. Can you elaborate on that a bit more? I can understand how > step-by-step would be potentially easier to implement, but does it also > capture the majority of expected application use cases? > > > > A good confirmation that this is the correct approach for v1.0 though. > > > > On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef > wrote: > > > > For what it's worth, as a SoC vendor rep. that has ordered queue in HW, > Petri's definition is actually preferred for us even going forward J > > > > > > Thanks, > > Gilad > > > >
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
Since odp_schedule_skip_order has dest queue, So how its differ from odp_queue_enque() as en-queue is going to release it implicitly. Jerin. From: lng-odp-boun...@lists.linaro.org on behalf of Alexandru Badicioiu Sent: Friday, October 31, 2014 6:40 PM To: Bill Fischofer Cc: Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions In this case the "dest" argument in the following function: +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf); looses it's meaning. Alex On 31 October 2014 14:59, Bill Fischofer mailto:bill.fischo...@linaro.org>> wrote: This may well be a reasonable restriction for ODP v1.0 but I believe it's something we need to put on the list for "production grade" improvements for 2015. Bill On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) mailto:petri.savolai...@nsn.com>> wrote: Yes, it’s step-by-step and I think it’s the level of ordering we need for v1.0. Most SoCs can implement it, even when the HW scheduler would not have order support but only atomic/parallel. This way defined atomic scheduling can be used to implement functionality correct ordered queues, the throughput is not improved but it functions correctly. -Petri From: ext Bill Fischofer [mailto:bill.fischo...@linaro.org<mailto:bill.fischo...@linaro.org>] Sent: Friday, October 31, 2014 2:48 PM To: Alexandru Badicioiu Cc: Petri Savolainen; lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions I can well imagine the step-by-step order preservation to be simpler to implement (in SW) but it would also seem to limit performance since the only way to ensure end-to-end order preservation would be if each intermediate queue from ingress to egress were an ordered queue. If there is a parallel queue anywhere in the chain that would break things. The question is: Is this restriction needed and/or sufficient for ODP v1.0? On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu mailto:alexandru.badici...@linaro.org>> wrote: "+ * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call" Is this assuming that the order will be restored always at the next enqueue? I think there should be an option to explicitly indicate if the next enqueue is supposed to restore the order or not, especially when packets move from queue to queue. Ordered queues are costly compared with the ordinary ones. Alex On 31 October 2014 14:25, Petri Savolainen mailto:petri.savolai...@linaro.org>> wrote: Improved atomic and ordered synchronisation definitions. Added order skip function prototype. Signed-off-by: Petri Savolainen mailto:petri.savolai...@linaro.org>> --- This is the ordered queue definition (in patch format) promised in the call yesterday. --- platform/linux-generic/include/api/odp_queue.h| 31 +++- platform/linux-generic/include/api/odp_schedule.h | 45 ++- 2 files changed, 64 insertions(+), 12 deletions(-) diff --git a/platform/linux-generic/include/api/odp_queue.h b/platform/linux-generic/include/api/odp_queue.h index b8ac4bb..c0c3969 100644 --- a/platform/linux-generic/include/api/odp_queue.h +++ b/platform/linux-generic/include/api/odp_queue.h @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; typedef int odp_schedule_sync_t; #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ + +/** + * Atomic queue synchronisation + * + * The scheduler gives buffers from a queue to a single core at a time. This + * serialises processing of the buffers from the source queue and helps user to + * avoid SW locking. Another schedule call will implicitely release the atomic + * synchronisation of the source queue and free the scheduler to give buffers + * from the queue to other cores. + * + * User can hint the scheduler to release the atomic synchronisation early with + * odp_schedule_release_atomic(). + */ +#define ODP_SCHED_SYNC_ATOMIC 1 + +/** + * Ordered queue synchronisation + * + * The scheduler may give out buffers to multiple cores for parallel processing. + * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call. Buffers + * from the same ordered (source) queue appear in their original order when + * dequeued from a destination queue. The destination queue type (POLL/SCHED) or + * synchronisation (NONE/ATOMIC/ORDERED) is not limited. + * + * User can command the scheduler to skip ordering of a buffer with + * odp_schedule_skip_order
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
I don't think it can use the first fragment place in the order. It can use the last fragment to arrive place in the queue. Otherwise you have to stop forwarding until all fragments arrive to keep the order… no? Gilad Ben-Yossef Software Architect EZchip Technologies Ltd. 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 Email: gil...@ezchip.com<mailto:gil...@ezchip.com>, Web: http://www.ezchip.com<http://www.ezchip.com/> "Ethernet always wins." — Andy Bechtolsheim From: Alexandru Badicioiu [mailto:alexandru.badici...@linaro.org] Sent: Tuesday, November 04, 2014 4:13 PM To: Bill Fischofer Cc: Gilad Ben Yossef; Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions In addition to skip order function I think we need to handle the case when a packet gets fragmented (e.g. IP fragmentation). In this case the fragments should share the original sequence number and the order restoration logic should be aware when the last fragment occurs. Alex On 3 November 2014 13:45, Bill Fischofer mailto:bill.fischo...@linaro.org>> wrote: OK, thanks. That certainly matches the step-by-step semantics Petri proposed. We'll have to discuss what, if anything, more we need regarding ORDERED queues post-v1.0 next year. Bill On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef mailto:gil...@ezchip.com>> wrote: What NPS does in HW is that each buffer you dequeue actually stays on the queue (but is marked as consumed by the application) and you are given a "tag" for it. When I want to queue the buffer to the next, I give the HW the tag I got, not the buffer handle and what the HW does is only than fully dequeue the buffer from the original queue before queuing it into the new queue – but only after all preceding (older) buffers on the original queue have been fully dequeued, Free of a buffer causes a dequeue from the original queue as well. And I can also give release a buffer from the original queue (give the HW the tag back and say "you don't have to wait for this buffer any longer). I hope this helps, Gilad Gilad Ben-Yossef Software Architect EZchip Technologies Ltd. 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 Email: gil...@ezchip.com<mailto:gil...@ezchip.com>, Web: http://www.ezchip.com<http://www.ezchip.com/> "Ethernet always wins." — Andy Bechtolsheim From: Bill Fischofer [mailto:bill.fischo...@linaro.org<mailto:bill.fischo...@linaro.org>] Sent: Sunday, November 02, 2014 5:44 PM To: Gilad Ben Yossef Cc: Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions Thanks Gilad. Can you elaborate on that a bit more? I can understand how step-by-step would be potentially easier to implement, but does it also capture the majority of expected application use cases? A good confirmation that this is the correct approach for v1.0 though. On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef mailto:gil...@ezchip.com>> wrote: For what it's worth, as a SoC vendor rep. that has ordered queue in HW, Petri's definition is actually preferred for us even going forward ☺ Thanks, Gilad Gilad Ben-Yossef Software Architect EZchip Technologies Ltd. 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 Email: gil...@ezchip.com<mailto:gil...@ezchip.com>, Web: http://www.ezchip.com<http://www.ezchip.com/> "Ethernet always wins." — Andy Bechtolsheim From: lng-odp-boun...@lists.linaro.org<mailto:lng-odp-boun...@lists.linaro.org> [mailto:lng-odp-boun...@lists.linaro.org<mailto:lng-odp-boun...@lists.linaro.org>] On Behalf Of Bill Fischofer Sent: Friday, October 31, 2014 2:59 PM To: Savolainen, Petri (NSN - FI/Espoo) Cc: lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions This may well be a reasonable restriction for ODP v1.0 but I believe it's something we need to put on the list for "production grade" improvements for 2015. Bill On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) mailto:petri.savolai...@nsn.com>> wrote: Yes, it’s step-by-step and I think it’s the level of ordering we need for v1.0. Most SoCs can implement it, even when the HW scheduler would not have order support but only atomic/parallel. This way defined atomic scheduling can be used to implement functionality correct ordered queues, the throughput is not
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
In addition to skip order function I think we need to handle the case when a packet gets fragmented (e.g. IP fragmentation). In this case the fragments should share the original sequence number and the order restoration logic should be aware when the last fragment occurs. Alex On 3 November 2014 13:45, Bill Fischofer wrote: > OK, thanks. That certainly matches the step-by-step semantics Petri > proposed. We'll have to discuss what, if anything, more we need regarding > ORDERED queues post-v1.0 next year. > > Bill > > On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef > wrote: > >> >> >> >> >> What NPS does in HW is that each buffer you dequeue actually stays on the >> queue (but is marked as consumed by the application) and you are given a >> "tag" for it. >> >> >> >> When I want to queue the buffer to the next, I give the HW the tag I got, >> not the buffer handle and what the HW does is only than fully dequeue the >> buffer from the original queue before queuing it into the new queue – but >> only after all preceding (older) buffers on the original queue have been >> fully dequeued, >> >> >> >> Free of a buffer causes a dequeue from the original queue as well. >> >> >> >> And I can also give release a buffer from the original queue (give the HW >> the tag back and say "you don't have to wait for this buffer any longer). >> >> >> >> I hope this helps, >> >> Gilad >> >> >> >> *Gilad Ben-Yossef* >> >> Software Architect >> >> EZchip Technologies Ltd. >> >> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel >> >> Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 >> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 >> >> Email: gil...@ezchip.com, Web: http://www.ezchip.com >> >> >> >> *"Ethernet always wins."* >> >> — Andy Bechtolsheim >> >> >> >> *From:* Bill Fischofer [mailto:bill.fischo...@linaro.org] >> *Sent:* Sunday, November 02, 2014 5:44 PM >> *To:* Gilad Ben Yossef >> *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org >> >> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions >> >> >> >> Thanks Gilad. Can you elaborate on that a bit more? I can understand >> how step-by-step would be potentially easier to implement, but does it also >> capture the majority of expected application use cases? >> >> >> >> A good confirmation that this is the correct approach for v1.0 though. >> >> >> >> On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef >> wrote: >> >> >> >> For what it's worth, as a SoC vendor rep. that has ordered queue in HW, >> Petri's definition is actually preferred for us even going forward J >> >> >> >> >> >> Thanks, >> >> Gilad >> >> >> >> *Gilad Ben-Yossef* >> >> Software Architect >> >> EZchip Technologies Ltd. >> >> 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel >> >> Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 >> Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 >> >> Email: gil...@ezchip.com, Web: http://www.ezchip.com >> >> >> >> *"Ethernet always wins."* >> >> — Andy Bechtolsheim >> >> >> >> *From:* lng-odp-boun...@lists.linaro.org [mailto: >> lng-odp-boun...@lists.linaro.org] *On Behalf Of *Bill Fischofer >> *Sent:* Friday, October 31, 2014 2:59 PM >> *To:* Savolainen, Petri (NSN - FI/Espoo) >> *Cc:* lng-odp@lists.linaro.org >> >> >> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions >> >> >> >> This may well be a reasonable restriction for ODP v1.0 but I believe it's >> something we need to put on the list for "production grade" improvements >> for 2015. >> >> >> >> Bill >> >> >> >> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) < >> petri.savolai...@nsn.com> wrote: >> >> Yes, it’s step-by-step and I think it’s the level of ordering we need for >> v1.0. Most SoCs can implement it, even when the HW scheduler would not >> have order support but only atomic/parallel. This way defined atomic >> scheduling can be used to implement functionality correct ordered queues, >> the throughput is not improved but it functions correctly. >> >&g
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
OK, thanks. That certainly matches the step-by-step semantics Petri proposed. We'll have to discuss what, if anything, more we need regarding ORDERED queues post-v1.0 next year. Bill On Mon, Nov 3, 2014 at 3:03 AM, Gilad Ben Yossef wrote: > > > > > What NPS does in HW is that each buffer you dequeue actually stays on the > queue (but is marked as consumed by the application) and you are given a > "tag" for it. > > > > When I want to queue the buffer to the next, I give the HW the tag I got, > not the buffer handle and what the HW does is only than fully dequeue the > buffer from the original queue before queuing it into the new queue – but > only after all preceding (older) buffers on the original queue have been > fully dequeued, > > > > Free of a buffer causes a dequeue from the original queue as well. > > > > And I can also give release a buffer from the original queue (give the HW > the tag back and say "you don't have to wait for this buffer any longer). > > > > I hope this helps, > > Gilad > > > > *Gilad Ben-Yossef* > > Software Architect > > EZchip Technologies Ltd. > > 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel > > Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 > Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 > > Email: gil...@ezchip.com, Web: http://www.ezchip.com > > > > *"Ethernet always wins."* > > — Andy Bechtolsheim > > > > *From:* Bill Fischofer [mailto:bill.fischo...@linaro.org] > *Sent:* Sunday, November 02, 2014 5:44 PM > *To:* Gilad Ben Yossef > *Cc:* Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org > > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > Thanks Gilad. Can you elaborate on that a bit more? I can understand how > step-by-step would be potentially easier to implement, but does it also > capture the majority of expected application use cases? > > > > A good confirmation that this is the correct approach for v1.0 though. > > > > On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef > wrote: > > > > For what it's worth, as a SoC vendor rep. that has ordered queue in HW, > Petri's definition is actually preferred for us even going forward J > > > > > > Thanks, > > Gilad > > > > *Gilad Ben-Yossef* > > Software Architect > > EZchip Technologies Ltd. > > 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel > > Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 > Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 > > Email: gil...@ezchip.com, Web: http://www.ezchip.com > > > > *"Ethernet always wins."* > > — Andy Bechtolsheim > > > > *From:* lng-odp-boun...@lists.linaro.org [mailto: > lng-odp-boun...@lists.linaro.org] *On Behalf Of *Bill Fischofer > *Sent:* Friday, October 31, 2014 2:59 PM > *To:* Savolainen, Petri (NSN - FI/Espoo) > *Cc:* lng-odp@lists.linaro.org > > > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > This may well be a reasonable restriction for ODP v1.0 but I believe it's > something we need to put on the list for "production grade" improvements > for 2015. > > > > Bill > > > > On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) < > petri.savolai...@nsn.com> wrote: > > Yes, it’s step-by-step and I think it’s the level of ordering we need for > v1.0. Most SoCs can implement it, even when the HW scheduler would not > have order support but only atomic/parallel. This way defined atomic > scheduling can be used to implement functionality correct ordered queues, > the throughput is not improved but it functions correctly. > > > > -Petri > > > > > > *From:* ext Bill Fischofer [mailto:bill.fischo...@linaro.org] > *Sent:* Friday, October 31, 2014 2:48 PM > *To:* Alexandru Badicioiu > *Cc:* Petri Savolainen; lng-odp@lists.linaro.org > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > I can well imagine the step-by-step order preservation to be simpler to > implement (in SW) but it would also seem to limit performance since the > only way to ensure end-to-end order preservation would be if each > intermediate queue from ingress to egress were an ordered queue. If there > is a parallel queue anywhere in the chain that would break things. > > > > The question is: Is this restriction needed and/or sufficient for ODP > v1.0? > > > > On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu < > alexandru.badici...@linaro.org> wrote: >
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
What NPS does in HW is that each buffer you dequeue actually stays on the queue (but is marked as consumed by the application) and you are given a "tag" for it. When I want to queue the buffer to the next, I give the HW the tag I got, not the buffer handle and what the HW does is only than fully dequeue the buffer from the original queue before queuing it into the new queue – but only after all preceding (older) buffers on the original queue have been fully dequeued, Free of a buffer causes a dequeue from the original queue as well. And I can also give release a buffer from the original queue (give the HW the tag back and say "you don't have to wait for this buffer any longer). I hope this helps, Gilad Gilad Ben-Yossef Software Architect EZchip Technologies Ltd. 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 Email: gil...@ezchip.com<mailto:gil...@ezchip.com>, Web: http://www.ezchip.com<http://www.ezchip.com/> "Ethernet always wins." — Andy Bechtolsheim From: Bill Fischofer [mailto:bill.fischo...@linaro.org] Sent: Sunday, November 02, 2014 5:44 PM To: Gilad Ben Yossef Cc: Savolainen, Petri (NSN - FI/Espoo); lng-odp@lists.linaro.org Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions Thanks Gilad. Can you elaborate on that a bit more? I can understand how step-by-step would be potentially easier to implement, but does it also capture the majority of expected application use cases? A good confirmation that this is the correct approach for v1.0 though. On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef mailto:gil...@ezchip.com>> wrote: For what it's worth, as a SoC vendor rep. that has ordered queue in HW, Petri's definition is actually preferred for us even going forward ☺ Thanks, Gilad Gilad Ben-Yossef Software Architect EZchip Technologies Ltd. 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 Email: gil...@ezchip.com<mailto:gil...@ezchip.com>, Web: http://www.ezchip.com<http://www.ezchip.com/> "Ethernet always wins." — Andy Bechtolsheim From: lng-odp-boun...@lists.linaro.org<mailto:lng-odp-boun...@lists.linaro.org> [mailto:lng-odp-boun...@lists.linaro.org<mailto:lng-odp-boun...@lists.linaro.org>] On Behalf Of Bill Fischofer Sent: Friday, October 31, 2014 2:59 PM To: Savolainen, Petri (NSN - FI/Espoo) Cc: lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions This may well be a reasonable restriction for ODP v1.0 but I believe it's something we need to put on the list for "production grade" improvements for 2015. Bill On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) mailto:petri.savolai...@nsn.com>> wrote: Yes, it’s step-by-step and I think it’s the level of ordering we need for v1.0. Most SoCs can implement it, even when the HW scheduler would not have order support but only atomic/parallel. This way defined atomic scheduling can be used to implement functionality correct ordered queues, the throughput is not improved but it functions correctly. -Petri From: ext Bill Fischofer [mailto:bill.fischo...@linaro.org<mailto:bill.fischo...@linaro.org>] Sent: Friday, October 31, 2014 2:48 PM To: Alexandru Badicioiu Cc: Petri Savolainen; lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions I can well imagine the step-by-step order preservation to be simpler to implement (in SW) but it would also seem to limit performance since the only way to ensure end-to-end order preservation would be if each intermediate queue from ingress to egress were an ordered queue. If there is a parallel queue anywhere in the chain that would break things. The question is: Is this restriction needed and/or sufficient for ODP v1.0? On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu mailto:alexandru.badici...@linaro.org>> wrote: "+ * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call" Is this assuming that the order will be restored always at the next enqueue? I think there should be an option to explicitly indicate if the next enqueue is supposed to restore the order or not, especially when packets move from queue to queue. Ordered queues are costly compared with the ordinary ones. Alex On 31 October 2014 14:25, Petri Savolainen mailto:petri.savolai...@linaro.org>> wrote: Improved atomic and ordered synchronisation definitions. Added order skip function prototype. Signed-off-by: Petri Savolainen mailto:petri.sa
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
Thanks Gilad. Can you elaborate on that a bit more? I can understand how step-by-step would be potentially easier to implement, but does it also capture the majority of expected application use cases? A good confirmation that this is the correct approach for v1.0 though. On Sun, Nov 2, 2014 at 9:26 AM, Gilad Ben Yossef wrote: > > > For what it's worth, as a SoC vendor rep. that has ordered queue in HW, > Petri's > definition is actually preferred for us even going forward J > > > > > > Thanks, > > Gilad > > > > *Gilad Ben-Yossef* > > Software Architect > > EZchip Technologies Ltd. > > 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel > > Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 > Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 > > Email: gil...@ezchip.com, Web: http://www.ezchip.com > > > > *"Ethernet always wins."* > > — Andy Bechtolsheim > > > > *From:* lng-odp-boun...@lists.linaro.org [mailto: > lng-odp-boun...@lists.linaro.org] *On Behalf Of *Bill Fischofer > *Sent:* Friday, October 31, 2014 2:59 PM > *To:* Savolainen, Petri (NSN - FI/Espoo) > *Cc:* lng-odp@lists.linaro.org > > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > This may well be a reasonable restriction for ODP v1.0 but I believe it's > something we need to put on the list for "production grade" improvements > for 2015. > > > > Bill > > > > On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) < > petri.savolai...@nsn.com> wrote: > > Yes, it’s step-by-step and I think it’s the level of ordering we need for > v1.0. Most SoCs can implement it, even when the HW scheduler would not > have order support but only atomic/parallel. This way defined atomic > scheduling can be used to implement functionality correct ordered queues, > the throughput is not improved but it functions correctly. > > > > -Petri > > > > > > *From:* ext Bill Fischofer [mailto:bill.fischo...@linaro.org] > *Sent:* Friday, October 31, 2014 2:48 PM > *To:* Alexandru Badicioiu > *Cc:* Petri Savolainen; lng-odp@lists.linaro.org > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > I can well imagine the step-by-step order preservation to be simpler to > implement (in SW) but it would also seem to limit performance since the > only way to ensure end-to-end order preservation would be if each > intermediate queue from ingress to egress were an ordered queue. If there > is a parallel queue anywhere in the chain that would break things. > > > > The question is: Is this restriction needed and/or sufficient for ODP > v1.0? > > > > On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu < > alexandru.badici...@linaro.org> wrote: > > "+ * The original enqueue order of the source queue is maintained when > buffers are > + * enqueued to their destination queue(s) before another schedule call" > > > > Is this assuming that the order will be restored always at the next > enqueue? I think there should be an option to explicitly indicate if the > next enqueue is supposed to restore the order or not, especially when > packets move from queue to queue. Ordered queues are costly compared with > the ordinary ones. > > > > Alex > > > > On 31 October 2014 14:25, Petri Savolainen > wrote: > > Improved atomic and ordered synchronisation definitions. Added > order skip function prototype. > > Signed-off-by: Petri Savolainen > > --- > This is the ordered queue definition (in patch format) promised > in the call yesterday. > --- > platform/linux-generic/include/api/odp_queue.h| 31 +++- > platform/linux-generic/include/api/odp_schedule.h | 45 > ++- > 2 files changed, 64 insertions(+), 12 deletions(-) > > diff --git a/platform/linux-generic/include/api/odp_queue.h > b/platform/linux-generic/include/api/odp_queue.h > index b8ac4bb..c0c3969 100644 > --- a/platform/linux-generic/include/api/odp_queue.h > +++ b/platform/linux-generic/include/api/odp_queue.h > @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; > typedef int odp_schedule_sync_t; > > #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ > -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ > -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ > + > +/** > + * Atomic queue synchronisation > + * > + * The scheduler gives buffers from a queue to a single core at a time. > This > + * serialises processing of the buffers from the source queue and helps > user to &g
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
For what it's worth, as a SoC vendor rep. that has ordered queue in HW, Petri's definition is actually preferred for us even going forward ☺ Thanks, Gilad Gilad Ben-Yossef Software Architect EZchip Technologies Ltd. 37 Israel Pollak Ave, Kiryat Gat 82025 ,Israel Tel: +972-4-959- ext. 576, Fax: +972-8-681-1483 Mobile: +972-52-826-0388, US Mobile: +1-973-826-0388 Email: gil...@ezchip.com<mailto:gil...@ezchip.com>, Web: http://www.ezchip.com<http://www.ezchip.com/> "Ethernet always wins." — Andy Bechtolsheim From: lng-odp-boun...@lists.linaro.org [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill Fischofer Sent: Friday, October 31, 2014 2:59 PM To: Savolainen, Petri (NSN - FI/Espoo) Cc: lng-odp@lists.linaro.org Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions This may well be a reasonable restriction for ODP v1.0 but I believe it's something we need to put on the list for "production grade" improvements for 2015. Bill On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) mailto:petri.savolai...@nsn.com>> wrote: Yes, it’s step-by-step and I think it’s the level of ordering we need for v1.0. Most SoCs can implement it, even when the HW scheduler would not have order support but only atomic/parallel. This way defined atomic scheduling can be used to implement functionality correct ordered queues, the throughput is not improved but it functions correctly. -Petri From: ext Bill Fischofer [mailto:bill.fischo...@linaro.org<mailto:bill.fischo...@linaro.org>] Sent: Friday, October 31, 2014 2:48 PM To: Alexandru Badicioiu Cc: Petri Savolainen; lng-odp@lists.linaro.org<mailto:lng-odp@lists.linaro.org> Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions I can well imagine the step-by-step order preservation to be simpler to implement (in SW) but it would also seem to limit performance since the only way to ensure end-to-end order preservation would be if each intermediate queue from ingress to egress were an ordered queue. If there is a parallel queue anywhere in the chain that would break things. The question is: Is this restriction needed and/or sufficient for ODP v1.0? On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu mailto:alexandru.badici...@linaro.org>> wrote: "+ * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call" Is this assuming that the order will be restored always at the next enqueue? I think there should be an option to explicitly indicate if the next enqueue is supposed to restore the order or not, especially when packets move from queue to queue. Ordered queues are costly compared with the ordinary ones. Alex On 31 October 2014 14:25, Petri Savolainen mailto:petri.savolai...@linaro.org>> wrote: Improved atomic and ordered synchronisation definitions. Added order skip function prototype. Signed-off-by: Petri Savolainen mailto:petri.savolai...@linaro.org>> --- This is the ordered queue definition (in patch format) promised in the call yesterday. --- platform/linux-generic/include/api/odp_queue.h| 31 +++- platform/linux-generic/include/api/odp_schedule.h | 45 ++- 2 files changed, 64 insertions(+), 12 deletions(-) diff --git a/platform/linux-generic/include/api/odp_queue.h b/platform/linux-generic/include/api/odp_queue.h index b8ac4bb..c0c3969 100644 --- a/platform/linux-generic/include/api/odp_queue.h +++ b/platform/linux-generic/include/api/odp_queue.h @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; typedef int odp_schedule_sync_t; #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ + +/** + * Atomic queue synchronisation + * + * The scheduler gives buffers from a queue to a single core at a time. This + * serialises processing of the buffers from the source queue and helps user to + * avoid SW locking. Another schedule call will implicitely release the atomic + * synchronisation of the source queue and free the scheduler to give buffers + * from the queue to other cores. + * + * User can hint the scheduler to release the atomic synchronisation early with + * odp_schedule_release_atomic(). + */ +#define ODP_SCHED_SYNC_ATOMIC 1 + +/** + * Ordered queue synchronisation + * + * The scheduler may give out buffers to multiple cores for parallel processing. + * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call. Buffers + * from the same ordered (source) queue appear in their original order when + * dequeued from a destination queue. The destination queue type (POLL/SCHED) or + * synchronisation (NONE/AT
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
It would be documentation in v1.0 but worth keeping since if the step-by-step restriction is relaxed next year it would mean we wouldn't have to change the API. On Fri, Oct 31, 2014 at 8:10 AM, Alexandru Badicioiu < alexandru.badici...@linaro.org> wrote: > In this case the "dest" argument in the following function: > +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf); > looses it's meaning. > > Alex > > > > On 31 October 2014 14:59, Bill Fischofer > wrote: > >> This may well be a reasonable restriction for ODP v1.0 but I believe it's >> something we need to put on the list for "production grade" improvements >> for 2015. >> >> Bill >> >> On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) < >> petri.savolai...@nsn.com> wrote: >> >>> Yes, it’s step-by-step and I think it’s the level of ordering we need >>> for v1.0. Most SoCs can implement it, even when the HW scheduler would not >>> have order support but only atomic/parallel. This way defined atomic >>> scheduling can be used to implement functionality correct ordered queues, >>> the throughput is not improved but it functions correctly. >>> >>> >>> >>> -Petri >>> >>> >>> >>> >>> >>> *From:* ext Bill Fischofer [mailto:bill.fischo...@linaro.org] >>> *Sent:* Friday, October 31, 2014 2:48 PM >>> *To:* Alexandru Badicioiu >>> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org >>> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered >>> definitions >>> >>> >>> >>> I can well imagine the step-by-step order preservation to be simpler to >>> implement (in SW) but it would also seem to limit performance since the >>> only way to ensure end-to-end order preservation would be if each >>> intermediate queue from ingress to egress were an ordered queue. If there >>> is a parallel queue anywhere in the chain that would break things. >>> >>> >>> >>> The question is: Is this restriction needed and/or sufficient for ODP >>> v1.0? >>> >>> >>> >>> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu < >>> alexandru.badici...@linaro.org> wrote: >>> >>> "+ * The original enqueue order of the source queue is maintained when >>> buffers are >>> + * enqueued to their destination queue(s) before another schedule call" >>> >>> >>> >>> Is this assuming that the order will be restored always at the next >>> enqueue? I think there should be an option to explicitly indicate if the >>> next enqueue is supposed to restore the order or not, especially when >>> packets move from queue to queue. Ordered queues are costly compared with >>> the ordinary ones. >>> >>> >>> >>> Alex >>> >>> >>> >>> On 31 October 2014 14:25, Petri Savolainen >>> wrote: >>> >>> Improved atomic and ordered synchronisation definitions. Added >>> order skip function prototype. >>> >>> Signed-off-by: Petri Savolainen >>> >>> --- >>> This is the ordered queue definition (in patch format) promised >>> in the call yesterday. >>> --- >>> platform/linux-generic/include/api/odp_queue.h| 31 +++- >>> platform/linux-generic/include/api/odp_schedule.h | 45 >>> ++- >>> 2 files changed, 64 insertions(+), 12 deletions(-) >>> >>> diff --git a/platform/linux-generic/include/api/odp_queue.h >>> b/platform/linux-generic/include/api/odp_queue.h >>> index b8ac4bb..c0c3969 100644 >>> --- a/platform/linux-generic/include/api/odp_queue.h >>> +++ b/platform/linux-generic/include/api/odp_queue.h >>> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; >>> typedef int odp_schedule_sync_t; >>> >>> #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ >>> -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ >>> -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ >>> + >>> +/** >>> + * Atomic queue synchronisation >>> + * >>> + * The scheduler gives buffers from a queue to a single core at a time. >>> This >>> + * serialises processing of the buffers from the source queue and helps >>> user to >>> + * avoid SW locking. Another schedu
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
In this case the "dest" argument in the following function: +void odp_schedule_skip_order(odp_queue_t dest, odp_buffer_t buf); looses it's meaning. Alex On 31 October 2014 14:59, Bill Fischofer wrote: > This may well be a reasonable restriction for ODP v1.0 but I believe it's > something we need to put on the list for "production grade" improvements > for 2015. > > Bill > > On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) < > petri.savolai...@nsn.com> wrote: > >> Yes, it’s step-by-step and I think it’s the level of ordering we need >> for v1.0. Most SoCs can implement it, even when the HW scheduler would not >> have order support but only atomic/parallel. This way defined atomic >> scheduling can be used to implement functionality correct ordered queues, >> the throughput is not improved but it functions correctly. >> >> >> >> -Petri >> >> >> >> >> >> *From:* ext Bill Fischofer [mailto:bill.fischo...@linaro.org] >> *Sent:* Friday, October 31, 2014 2:48 PM >> *To:* Alexandru Badicioiu >> *Cc:* Petri Savolainen; lng-odp@lists.linaro.org >> *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions >> >> >> >> I can well imagine the step-by-step order preservation to be simpler to >> implement (in SW) but it would also seem to limit performance since the >> only way to ensure end-to-end order preservation would be if each >> intermediate queue from ingress to egress were an ordered queue. If there >> is a parallel queue anywhere in the chain that would break things. >> >> >> >> The question is: Is this restriction needed and/or sufficient for ODP >> v1.0? >> >> >> >> On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu < >> alexandru.badici...@linaro.org> wrote: >> >> "+ * The original enqueue order of the source queue is maintained when >> buffers are >> + * enqueued to their destination queue(s) before another schedule call" >> >> >> >> Is this assuming that the order will be restored always at the next >> enqueue? I think there should be an option to explicitly indicate if the >> next enqueue is supposed to restore the order or not, especially when >> packets move from queue to queue. Ordered queues are costly compared with >> the ordinary ones. >> >> >> >> Alex >> >> >> >> On 31 October 2014 14:25, Petri Savolainen >> wrote: >> >> Improved atomic and ordered synchronisation definitions. Added >> order skip function prototype. >> >> Signed-off-by: Petri Savolainen >> >> --- >> This is the ordered queue definition (in patch format) promised >> in the call yesterday. >> --- >> platform/linux-generic/include/api/odp_queue.h| 31 +++- >> platform/linux-generic/include/api/odp_schedule.h | 45 >> ++- >> 2 files changed, 64 insertions(+), 12 deletions(-) >> >> diff --git a/platform/linux-generic/include/api/odp_queue.h >> b/platform/linux-generic/include/api/odp_queue.h >> index b8ac4bb..c0c3969 100644 >> --- a/platform/linux-generic/include/api/odp_queue.h >> +++ b/platform/linux-generic/include/api/odp_queue.h >> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; >> typedef int odp_schedule_sync_t; >> >> #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ >> -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ >> -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ >> + >> +/** >> + * Atomic queue synchronisation >> + * >> + * The scheduler gives buffers from a queue to a single core at a time. >> This >> + * serialises processing of the buffers from the source queue and helps >> user to >> + * avoid SW locking. Another schedule call will implicitely release the >> atomic >> + * synchronisation of the source queue and free the scheduler to give >> buffers >> + * from the queue to other cores. >> + * >> + * User can hint the scheduler to release the atomic synchronisation >> early with >> + * odp_schedule_release_atomic(). >> + */ >> +#define ODP_SCHED_SYNC_ATOMIC 1 >> + >> +/** >> + * Ordered queue synchronisation >> + * >> + * The scheduler may give out buffers to multiple cores for parallel >> processing. >> + * The original enqueue order of the source queue is maintained when >> buffers are >> + * enqueued to their destination queue(s) before ano
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
This may well be a reasonable restriction for ODP v1.0 but I believe it's something we need to put on the list for "production grade" improvements for 2015. Bill On Fri, Oct 31, 2014 at 7:57 AM, Savolainen, Petri (NSN - FI/Espoo) < petri.savolai...@nsn.com> wrote: > Yes, it’s step-by-step and I think it’s the level of ordering we need > for v1.0. Most SoCs can implement it, even when the HW scheduler would not > have order support but only atomic/parallel. This way defined atomic > scheduling can be used to implement functionality correct ordered queues, > the throughput is not improved but it functions correctly. > > > > -Petri > > > > > > *From:* ext Bill Fischofer [mailto:bill.fischo...@linaro.org] > *Sent:* Friday, October 31, 2014 2:48 PM > *To:* Alexandru Badicioiu > *Cc:* Petri Savolainen; lng-odp@lists.linaro.org > *Subject:* Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions > > > > I can well imagine the step-by-step order preservation to be simpler to > implement (in SW) but it would also seem to limit performance since the > only way to ensure end-to-end order preservation would be if each > intermediate queue from ingress to egress were an ordered queue. If there > is a parallel queue anywhere in the chain that would break things. > > > > The question is: Is this restriction needed and/or sufficient for ODP > v1.0? > > > > On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu < > alexandru.badici...@linaro.org> wrote: > > "+ * The original enqueue order of the source queue is maintained when > buffers are > + * enqueued to their destination queue(s) before another schedule call" > > > > Is this assuming that the order will be restored always at the next > enqueue? I think there should be an option to explicitly indicate if the > next enqueue is supposed to restore the order or not, especially when > packets move from queue to queue. Ordered queues are costly compared with > the ordinary ones. > > > > Alex > > > > On 31 October 2014 14:25, Petri Savolainen > wrote: > > Improved atomic and ordered synchronisation definitions. Added > order skip function prototype. > > Signed-off-by: Petri Savolainen > > --- > This is the ordered queue definition (in patch format) promised > in the call yesterday. > --- > platform/linux-generic/include/api/odp_queue.h| 31 +++- > platform/linux-generic/include/api/odp_schedule.h | 45 > ++- > 2 files changed, 64 insertions(+), 12 deletions(-) > > diff --git a/platform/linux-generic/include/api/odp_queue.h > b/platform/linux-generic/include/api/odp_queue.h > index b8ac4bb..c0c3969 100644 > --- a/platform/linux-generic/include/api/odp_queue.h > +++ b/platform/linux-generic/include/api/odp_queue.h > @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; > typedef int odp_schedule_sync_t; > > #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ > -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ > -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ > + > +/** > + * Atomic queue synchronisation > + * > + * The scheduler gives buffers from a queue to a single core at a time. > This > + * serialises processing of the buffers from the source queue and helps > user to > + * avoid SW locking. Another schedule call will implicitely release the > atomic > + * synchronisation of the source queue and free the scheduler to give > buffers > + * from the queue to other cores. > + * > + * User can hint the scheduler to release the atomic synchronisation > early with > + * odp_schedule_release_atomic(). > + */ > +#define ODP_SCHED_SYNC_ATOMIC 1 > + > +/** > + * Ordered queue synchronisation > + * > + * The scheduler may give out buffers to multiple cores for parallel > processing. > + * The original enqueue order of the source queue is maintained when > buffers are > + * enqueued to their destination queue(s) before another schedule call. > Buffers > + * from the same ordered (source) queue appear in their original order > when > + * dequeued from a destination queue. The destination queue type > (POLL/SCHED) or > + * synchronisation (NONE/ATOMIC/ORDERED) is not limited. > + * > + * User can command the scheduler to skip ordering of a buffer with > + * odp_schedule_skip_order(). > + */ > +#define ODP_SCHED_SYNC_ORDERED 2 > > /** Default queue synchronisation */ > #define ODP_SCHED_SYNC_DEFAULT ODP_SCHED_SYNC_ATOMIC > diff --git a/platform/linux-generic/include/api/odp_schedule.h > b/platform/linux-generic/include/api/odp_schedule.h > index 91fec10..2a1a642 100644
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
Yes, it’s step-by-step and I think it’s the level of ordering we need for v1.0. Most SoCs can implement it, even when the HW scheduler would not have order support but only atomic/parallel. This way defined atomic scheduling can be used to implement functionality correct ordered queues, the throughput is not improved but it functions correctly. -Petri From: ext Bill Fischofer [mailto:bill.fischo...@linaro.org] Sent: Friday, October 31, 2014 2:48 PM To: Alexandru Badicioiu Cc: Petri Savolainen; lng-odp@lists.linaro.org Subject: Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions I can well imagine the step-by-step order preservation to be simpler to implement (in SW) but it would also seem to limit performance since the only way to ensure end-to-end order preservation would be if each intermediate queue from ingress to egress were an ordered queue. If there is a parallel queue anywhere in the chain that would break things. The question is: Is this restriction needed and/or sufficient for ODP v1.0? On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu mailto:alexandru.badici...@linaro.org>> wrote: "+ * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call" Is this assuming that the order will be restored always at the next enqueue? I think there should be an option to explicitly indicate if the next enqueue is supposed to restore the order or not, especially when packets move from queue to queue. Ordered queues are costly compared with the ordinary ones. Alex On 31 October 2014 14:25, Petri Savolainen mailto:petri.savolai...@linaro.org>> wrote: Improved atomic and ordered synchronisation definitions. Added order skip function prototype. Signed-off-by: Petri Savolainen mailto:petri.savolai...@linaro.org>> --- This is the ordered queue definition (in patch format) promised in the call yesterday. --- platform/linux-generic/include/api/odp_queue.h| 31 +++- platform/linux-generic/include/api/odp_schedule.h | 45 ++- 2 files changed, 64 insertions(+), 12 deletions(-) diff --git a/platform/linux-generic/include/api/odp_queue.h b/platform/linux-generic/include/api/odp_queue.h index b8ac4bb..c0c3969 100644 --- a/platform/linux-generic/include/api/odp_queue.h +++ b/platform/linux-generic/include/api/odp_queue.h @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; typedef int odp_schedule_sync_t; #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ + +/** + * Atomic queue synchronisation + * + * The scheduler gives buffers from a queue to a single core at a time. This + * serialises processing of the buffers from the source queue and helps user to + * avoid SW locking. Another schedule call will implicitely release the atomic + * synchronisation of the source queue and free the scheduler to give buffers + * from the queue to other cores. + * + * User can hint the scheduler to release the atomic synchronisation early with + * odp_schedule_release_atomic(). + */ +#define ODP_SCHED_SYNC_ATOMIC 1 + +/** + * Ordered queue synchronisation + * + * The scheduler may give out buffers to multiple cores for parallel processing. + * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call. Buffers + * from the same ordered (source) queue appear in their original order when + * dequeued from a destination queue. The destination queue type (POLL/SCHED) or + * synchronisation (NONE/ATOMIC/ORDERED) is not limited. + * + * User can command the scheduler to skip ordering of a buffer with + * odp_schedule_skip_order(). + */ +#define ODP_SCHED_SYNC_ORDERED 2 /** Default queue synchronisation */ #define ODP_SCHED_SYNC_DEFAULT ODP_SCHED_SYNC_ATOMIC diff --git a/platform/linux-generic/include/api/odp_schedule.h b/platform/linux-generic/include/api/odp_schedule.h index 91fec10..2a1a642 100644 --- a/platform/linux-generic/include/api/odp_schedule.h +++ b/platform/linux-generic/include/api/odp_schedule.h @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns); * for a buffer according to the wait parameter setting. Returns * ODP_BUFFER_INVALID if reaches end of the wait period. * - * @param fromOutput parameter for the source queue (where the buffer was - *dequeued from). Ignored if NULL. + * @param src The source queue (output). Indicates from which queue the + *buffer was dequeued. Ignored if NULL. * @param waitMinimum time to wait for a buffer. Waits infinitely, if set to *ODP_SCHED_WAIT. Does not wait, if set to ODP_SCHED_NO_WAIT. *Use odp_schedule_wait_time() to convert time to other wait @@ -61,7 +61
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
I can well imagine the step-by-step order preservation to be simpler to implement (in SW) but it would also seem to limit performance since the only way to ensure end-to-end order preservation would be if each intermediate queue from ingress to egress were an ordered queue. If there is a parallel queue anywhere in the chain that would break things. The question is: Is this restriction needed and/or sufficient for ODP v1.0? On Fri, Oct 31, 2014 at 7:42 AM, Alexandru Badicioiu < alexandru.badici...@linaro.org> wrote: > "+ * The original enqueue order of the source queue is maintained when > buffers are > + * enqueued to their destination queue(s) before another schedule call" > > Is this assuming that the order will be restored always at the next > enqueue? I think there should be an option to explicitly indicate if the > next enqueue is supposed to restore the order or not, especially when > packets move from queue to queue. Ordered queues are costly compared with > the ordinary ones. > > Alex > > On 31 October 2014 14:25, Petri Savolainen > wrote: > >> Improved atomic and ordered synchronisation definitions. Added >> order skip function prototype. >> >> Signed-off-by: Petri Savolainen >> >> --- >> This is the ordered queue definition (in patch format) promised >> in the call yesterday. >> --- >> platform/linux-generic/include/api/odp_queue.h| 31 +++- >> platform/linux-generic/include/api/odp_schedule.h | 45 >> ++- >> 2 files changed, 64 insertions(+), 12 deletions(-) >> >> diff --git a/platform/linux-generic/include/api/odp_queue.h >> b/platform/linux-generic/include/api/odp_queue.h >> index b8ac4bb..c0c3969 100644 >> --- a/platform/linux-generic/include/api/odp_queue.h >> +++ b/platform/linux-generic/include/api/odp_queue.h >> @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; >> typedef int odp_schedule_sync_t; >> >> #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ >> -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ >> -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ >> + >> +/** >> + * Atomic queue synchronisation >> + * >> + * The scheduler gives buffers from a queue to a single core at a time. >> This >> + * serialises processing of the buffers from the source queue and helps >> user to >> + * avoid SW locking. Another schedule call will implicitely release the >> atomic >> + * synchronisation of the source queue and free the scheduler to give >> buffers >> + * from the queue to other cores. >> + * >> + * User can hint the scheduler to release the atomic synchronisation >> early with >> + * odp_schedule_release_atomic(). >> + */ >> +#define ODP_SCHED_SYNC_ATOMIC 1 >> + >> +/** >> + * Ordered queue synchronisation >> + * >> + * The scheduler may give out buffers to multiple cores for parallel >> processing. >> + * The original enqueue order of the source queue is maintained when >> buffers are >> + * enqueued to their destination queue(s) before another schedule call. >> Buffers >> + * from the same ordered (source) queue appear in their original order >> when >> + * dequeued from a destination queue. The destination queue type >> (POLL/SCHED) or >> + * synchronisation (NONE/ATOMIC/ORDERED) is not limited. >> + * >> + * User can command the scheduler to skip ordering of a buffer with >> + * odp_schedule_skip_order(). >> + */ >> +#define ODP_SCHED_SYNC_ORDERED 2 >> >> /** Default queue synchronisation */ >> #define ODP_SCHED_SYNC_DEFAULT ODP_SCHED_SYNC_ATOMIC >> diff --git a/platform/linux-generic/include/api/odp_schedule.h >> b/platform/linux-generic/include/api/odp_schedule.h >> index 91fec10..2a1a642 100644 >> --- a/platform/linux-generic/include/api/odp_schedule.h >> +++ b/platform/linux-generic/include/api/odp_schedule.h >> @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns); >> * for a buffer according to the wait parameter setting. Returns >> * ODP_BUFFER_INVALID if reaches end of the wait period. >> * >> - * @param fromOutput parameter for the source queue (where the >> buffer was >> - *dequeued from). Ignored if NULL. >> + * @param src The source queue (output). Indicates from which queue >> the >> + *buffer was dequeued. Ignored if NULL. >> * @param waitMinimum time to wait for a buffer. Waits infinitely, >> if set to >> *ODP_SCHED_WAIT. Does not wait, if set to >> ODP_SCHED_NO_WAIT. >> *Use odp_schedule_wait_time() to convert time to other >> wait >> @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns); >> * >> * @return Next highest priority buffer, or ODP_BUFFER_INVALID >> */ >> -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait); >> +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait); >> >> /** >> * Schedule one buffer >> @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t >> wait); >> * >> * User can exit the schedule loop without first calling
Re: [lng-odp] [PATCH] Scheduler atomic and ordered definitions
"+ * The original enqueue order of the source queue is maintained when buffers are + * enqueued to their destination queue(s) before another schedule call" Is this assuming that the order will be restored always at the next enqueue? I think there should be an option to explicitly indicate if the next enqueue is supposed to restore the order or not, especially when packets move from queue to queue. Ordered queues are costly compared with the ordinary ones. Alex On 31 October 2014 14:25, Petri Savolainen wrote: > Improved atomic and ordered synchronisation definitions. Added > order skip function prototype. > > Signed-off-by: Petri Savolainen > > --- > This is the ordered queue definition (in patch format) promised > in the call yesterday. > --- > platform/linux-generic/include/api/odp_queue.h| 31 +++- > platform/linux-generic/include/api/odp_schedule.h | 45 > ++- > 2 files changed, 64 insertions(+), 12 deletions(-) > > diff --git a/platform/linux-generic/include/api/odp_queue.h > b/platform/linux-generic/include/api/odp_queue.h > index b8ac4bb..c0c3969 100644 > --- a/platform/linux-generic/include/api/odp_queue.h > +++ b/platform/linux-generic/include/api/odp_queue.h > @@ -78,8 +78,35 @@ typedef int odp_schedule_prio_t; > typedef int odp_schedule_sync_t; > > #define ODP_SCHED_SYNC_NONE 0 /**< Queue not synchronised */ > -#define ODP_SCHED_SYNC_ATOMIC 1 /**< Atomic queue */ > -#define ODP_SCHED_SYNC_ORDERED 2 /**< Ordered queue */ > + > +/** > + * Atomic queue synchronisation > + * > + * The scheduler gives buffers from a queue to a single core at a time. > This > + * serialises processing of the buffers from the source queue and helps > user to > + * avoid SW locking. Another schedule call will implicitely release the > atomic > + * synchronisation of the source queue and free the scheduler to give > buffers > + * from the queue to other cores. > + * > + * User can hint the scheduler to release the atomic synchronisation > early with > + * odp_schedule_release_atomic(). > + */ > +#define ODP_SCHED_SYNC_ATOMIC 1 > + > +/** > + * Ordered queue synchronisation > + * > + * The scheduler may give out buffers to multiple cores for parallel > processing. > + * The original enqueue order of the source queue is maintained when > buffers are > + * enqueued to their destination queue(s) before another schedule call. > Buffers > + * from the same ordered (source) queue appear in their original order > when > + * dequeued from a destination queue. The destination queue type > (POLL/SCHED) or > + * synchronisation (NONE/ATOMIC/ORDERED) is not limited. > + * > + * User can command the scheduler to skip ordering of a buffer with > + * odp_schedule_skip_order(). > + */ > +#define ODP_SCHED_SYNC_ORDERED 2 > > /** Default queue synchronisation */ > #define ODP_SCHED_SYNC_DEFAULT ODP_SCHED_SYNC_ATOMIC > diff --git a/platform/linux-generic/include/api/odp_schedule.h > b/platform/linux-generic/include/api/odp_schedule.h > index 91fec10..2a1a642 100644 > --- a/platform/linux-generic/include/api/odp_schedule.h > +++ b/platform/linux-generic/include/api/odp_schedule.h > @@ -52,8 +52,8 @@ uint64_t odp_schedule_wait_time(uint64_t ns); > * for a buffer according to the wait parameter setting. Returns > * ODP_BUFFER_INVALID if reaches end of the wait period. > * > - * @param fromOutput parameter for the source queue (where the buffer > was > - *dequeued from). Ignored if NULL. > + * @param src The source queue (output). Indicates from which queue > the > + *buffer was dequeued. Ignored if NULL. > * @param waitMinimum time to wait for a buffer. Waits infinitely, if > set to > *ODP_SCHED_WAIT. Does not wait, if set to > ODP_SCHED_NO_WAIT. > *Use odp_schedule_wait_time() to convert time to other > wait > @@ -61,7 +61,7 @@ uint64_t odp_schedule_wait_time(uint64_t ns); > * > * @return Next highest priority buffer, or ODP_BUFFER_INVALID > */ > -odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t wait); > +odp_buffer_t odp_schedule(odp_queue_t *src, uint64_t wait); > > /** > * Schedule one buffer > @@ -76,8 +76,8 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t > wait); > * > * User can exit the schedule loop without first calling > odp_schedule_pause(). > * > - * @param fromOutput parameter for the source queue (where the buffer > was > - *dequeued from). Ignored if NULL. > + * @param src The source queue (output). Indicates from which queue > the > + *buffer was dequeued. Ignored if NULL. > * @param waitMinimum time to wait for a buffer. Waits infinitely, if > set to > *ODP_SCHED_WAIT. Does not wait, if set to > ODP_SCHED_NO_WAIT. > *Use odp_schedule_wait_time() to convert time to other > wait > @@ -85,7 +85,7 @@ odp_buffer_t odp_schedule(odp_queue_t *from, uint64_t > wait); > * > *
