On 6/29/20 8:37 AM, Lai Jiangshan wrote:
> On Mon, Jun 29, 2020 at 8:13 AM Bob Liu <bob....@oracle.com> wrote:
>>
>> On 6/28/20 11:54 PM, Lai Jiangshan wrote:
>>> On Thu, Jun 11, 2020 at 6:29 PM Bob Liu <bob....@oracle.com> wrote:
>>>>
>>>> Current code always set 'Unbound && max_active == 1' workqueues to ordered
>>>> implicitly, while this may be not an expected behaviour for some use cases.
>>>>
>>>> E.g some scsi and iscsi workqueues(unbound && max_active = 1) want to be
>>>> bind
>>>> to different cpu so as to get better isolation, but their cpumask can't be
>>>> changed because WQ_ORDERED is set implicitly.
>>>
>>> Hello
>>>
>>> If I read the code correctly, the reason why their cpumask can't
>>> be changed is because __WQ_ORDERED_EXPLICIT, not __WQ_ORDERED.
>>>
>>>>
>>>> This patch adds a flag __WQ_ORDERED_DISABLE and also
>>>> create_singlethread_workqueue_noorder() to offer an new option.
>>>>
>>>> Signed-off-by: Bob Liu <bob....@oracle.com>
>>>> ---
>>>> include/linux/workqueue.h | 4 ++++
>>>> kernel/workqueue.c | 4 +++-
>>>> 2 files changed, 7 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
>>>> index e48554e..4c86913 100644
>>>> --- a/include/linux/workqueue.h
>>>> +++ b/include/linux/workqueue.h
>>>> @@ -344,6 +344,7 @@ enum {
>>>> __WQ_ORDERED = 1 << 17, /* internal: workqueue is
>>>> ordered */
>>>> __WQ_LEGACY = 1 << 18, /* internal:
>>>> create*_workqueue() */
>>>> __WQ_ORDERED_EXPLICIT = 1 << 19, /* internal:
>>>> alloc_ordered_workqueue() */
>>>> + __WQ_ORDERED_DISABLE = 1 << 20, /* internal: don't set
>>>> __WQ_ORDERED implicitly */
>>>>
>>>> WQ_MAX_ACTIVE = 512, /* I like 512, better ideas? */
>>>> WQ_MAX_UNBOUND_PER_CPU = 4, /* 4 * #cpus for unbound wq */
>>>> @@ -433,6 +434,9 @@ struct workqueue_struct *alloc_workqueue(const char
>>>> *fmt,
>>>> #define create_singlethread_workqueue(name) \
>>>> alloc_ordered_workqueue("%s", __WQ_LEGACY | WQ_MEM_RECLAIM, name)
>>>>
>>>> +#define create_singlethread_workqueue_noorder(name) \
>>>> + alloc_workqueue("%s", WQ_SYSFS | __WQ_LEGACY | WQ_MEM_RECLAIM | \
>>>> + WQ_UNBOUND | __WQ_ORDERED_DISABLE, 1, (name))
>>>
>>> I think using __WQ_ORDERED without __WQ_ORDERED_EXPLICIT is what you
>>> need, in which case cpumask is allowed to be changed.
>>>
>>
>> I don't think so, see function workqueue_apply_unbound_cpumask():
>>
>> wq_unbound_cpumask_store()
>> > workqueue_set_unbound_cpumask()
>> > workqueue_apply_unbound_cpumask() {
>> ...
>> 5276 /* creating multiple pwqs breaks ordering guarantee */
>> 5277 if (wq->flags & __WQ_ORDERED)
>> 5278 continue;
>> ^^^^
>> Here will skip apply cpumask if only __WQ_ORDERED
>> is set.
>
> wq_unbound_cpumask_store() is for changing the cpumask of
> *all* workqueues. I don't think it can be used to make
> scsi and iscsi workqueues bound to different cpu.
>
> apply_workqueue_attrs() is for changing the cpumask of the specific
> workqueue, which can change the cpumask of __WQ_ORDERED workqueue
> (but without __WQ_ORDERED_EXPLICIT).
>
Yes, you are right. I made a mistake.
Sorry for the noise.
Regards,
Bob
>>
>> 5280 ctx = apply_wqattrs_prepare(wq, wq->unbound_attrs);
>>
>> }
