Best Regards,
Stanislaw Kardach
On 01/19/2017 04:57 PM, Bill Fischofer wrote:
On Thu, Jan 19, 2017 at 7:17 AM, Stanislaw Kardach <k...@semihalf.com> wrote:
Hi all,
While going through thread and scheduler APIs I've stumbled on one
uncertainty in ODP API that I could not find straight solution to.
Does ODP allow scheduling packets from an ODP_THREAD_CONTROL thread?
When a thread calls odp_schedule() the only requirement is that it be
a member of a schedule group that contains queues of type
ODP_QUEUE_TYPE_SCHED. Since threads by default are in group
ODP_SCHED_GROUP_ALL and that's also the default scheduler group for
queues, this is normally not a consideration.
If yes then what would be the difference between ODP_THREAD_CONTROL and
ODP_THREAD_WORKER threads beside isolating cores for worker threads? API
suggests this approach as both control and worker threads are treated the
same way in odp_thrmask_* calls. Moreover:
a. schedule groups take odp_thrmask_t (no comment on whether it has
to only contain worker threads)
b. There is a schedule group ODP_SCHED_GROUP_ALL which would imply that
user can create a scheduler queue that control threads can use.
The provision of WORKER/CONTROL threads is for application convenience
in organizing itself. But there is no semantic difference between the
two as far as the ODP API spec is concerned.
On the other hand I can find the following in ODP_THREAD_CONTROL
description: "Control threads do not participate the main packet flow
through the system". That looks to me like an implication that control
threads should not do any packet processing. However if that's the case then
ODP_THREAD_COUNT_MAX does not differentiate between worker or control
threads and similarly odp_thrmask_t doesn't (and by extension schedule
groups).
The expectation here is that worker threads will want to run on
isolated cores for best performance while control thread can share
cores and be timesliced without performance impact. That's the main
reason for having this division. Threads that do performance-critical
work would normally be designated worker thread while those that do
less performance-critical work would be designated as control threads.
Again, this is a convenience feature that applications can use to
manage core/thread assignments but ODP imposes no requirements on
applications in this area.
To put this discussion in a concrete context, on the platform which I'm
working on, each thread that wants to interact with a scheduler needs to do
it via special hardware handle of which I have a limited number. For me it
makes sense to reserve such handles only for threads which are going to do
the traffic processing (hence worker threads) and leave control threads
unlimited. In summary let application spawn as many control threads as it
wants but limit worker threads by the amount of handles that I have to
spare.
That's certainly one possibility. On such platforms the odp_schedule()
API might take the thread type into consideration in determining how
to process schedule requests, however in the case you outline a better
and more portable way might be to have a hardware-queue schedule group
and assign queues that are holding performance-critical events to that
schedule group. The point is that both applications and ODP
implementations have a lot of flexibility in how they operate within
the ODP API spec.
Do I understand you correctly that this hardware-queue schedule group
would be the one to utilize the "hardware handles" (and hence hardware
scheduler) where as other schedule groups rely on software scheduler?
--
Best Regards,
Stanislaw Kardach
