RE: [PATCH] net: fec: select queue depending on VLAN priority
From: Stefan Agner Sent: Monday, May 15, 2017 1:39 PM
>To: Andy Duan
>Cc: David Miller ; and...@lunn.ch;
>feste...@gmail.com; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>
>On 2017-05-10 21:49, Andy Duan wrote:
>> From: Stefan Agner Sent: Thursday, May 11, 2017
>> 12:08 PM
>>>To: Andy Duan
>>>Cc: David Miller ; and...@lunn.ch;
>>>feste...@gmail.com; net...@vger.kernel.org; linux-
>>>ker...@vger.kernel.org
>>>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>>>
>>>On 2017-05-09 19:42, Andy Duan wrote:
>>>> From: David Miller Sent: Tuesday, May 09, 2017
>>>> 9:39 PM
>>>>>To: ste...@agner.ch
>>>>>Cc: Andy Duan ; and...@lunn.ch;
>>>>>feste...@gmail.com; net...@vger.kernel.org; linux-
>>>>>ker...@vger.kernel.org
>>>>>Subject: Re: [PATCH] net: fec: select queue depending on VLAN
>>>>>priority
>>>>>
>>>>>From: Stefan Agner
>>>>>Date: Mon, 8 May 2017 22:37:08 -0700
>>>>>
>>>>>> Since the addition of the multi queue code with commit
>>>>>> 59d0f7465644
>>>>>> ("net: fec: init multi queue date structure") the queue selection
>>>>>> has been handelt by the default transmit queue selection
>>>>>> implementation which tries to evenly distribute the traffic across
>>>>>> all available queues. This selection presumes that the queues are
>>>>>> using an equal priority, however, the queues 1 and 2 are actually
>>>>>> of higher priority (the classification of the queues is enabled in
>>>fec_enet_enable_ring).
>>>>>>
>>>>>> This can lead to net scheduler warnings and continuous TX ring
>>>>>> dumps when exercising the system with iperf.
>>>>>>
>>>>>> Use only queue 0 for all common traffic (no VLAN and P802.1p
>>>>>> priority
>>>>>> 0 and 1) and route level 2-7 through queue 1 and 2.
>>>>>>
>>>>>> Signed-off-by: Fugang Duan
>>>>>> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
>>>>>
>>>>>If the queues are used for prioritization, and it does not have
>>>>>multiple normal priority level queues, multiqueue is not what the
>>>>>driver should have implemented.
>>>> Firstly, HW multiple queues support:
>>>>- Traffic-shaping bandwidth distribution supports credit-based and
>>>> round-robin-based policies. Either policy can be combined with
>>>> time-based shaping.
>>>>- AVB (Audio Video Bridging, IEEE 802.1Qav) features:
>>>>* Credit-based bandwidth distribution policy can be combined
>>>with
>>>> time-based shaping
>>>>* AVB endpoint talker and listener support
>>>>* Support for arbitration between different priority traffic
>>>> (for
>>>> example, AVB class A, AVB class B, and non-AVB) Round-robin-based
>>>> policies:
>>>>It has the same priority for three queues: In the round-robin QoS
>>>> scheme, each queue is given an equal opportunity to transmit one
>>>> frame. For example, if queue n has a frame to transmit, the queue
>>>> transmits its frame. After queue n has transmitted its frame, or if
>>>> queue n does not have a frame to transmit, queue n+1 is then allowed
>>>> to transmit its frame, and so on.
>>>>
>>>> Credit-based policies:
>>>>The AVB credit based shaper acts independently, per class, to
>>>> control the bandwidth distribution between normal traffic and
>>>> time-sensitive traffic with respect to the total link bandwidth available.
>>>>Credit based shaper conbined with time-based shaping:
>>>>- priority: ClassA queue > ClassB queue > best effort
>>>>- ensure the queue bandwidth as user set based on time-
>>>based shaping
>>>> algorithms (transmitter transmit frame from three queue in turn
>>>> based on time-based shaping algorithms)
>>>>And in real AVB case, each streaming can be independent, and are
>>>> fixed on related queue. Then driver level should implement
>>>> .ndo_select_queue() to put the streaming into related queue. That
Re: [PATCH] net: fec: select queue depending on VLAN priority
On 2017-05-10 21:49, Andy Duan wrote:
> From: Stefan Agner Sent: Thursday, May 11, 2017 12:08 PM
>>To: Andy Duan
>>Cc: David Miller ; and...@lunn.ch;
>>feste...@gmail.com; net...@vger.kernel.org; linux-
>>ker...@vger.kernel.org
>>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>>
>>On 2017-05-09 19:42, Andy Duan wrote:
>>> From: David Miller Sent: Tuesday, May 09, 2017
>>> 9:39 PM
>>>>To: ste...@agner.ch
>>>>Cc: Andy Duan ; and...@lunn.ch;
>>>>feste...@gmail.com; net...@vger.kernel.org; linux-
>>>>ker...@vger.kernel.org
>>>>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>>>>
>>>>From: Stefan Agner
>>>>Date: Mon, 8 May 2017 22:37:08 -0700
>>>>
>>>>> Since the addition of the multi queue code with commit 59d0f7465644
>>>>> ("net: fec: init multi queue date structure") the queue selection
>>>>> has been handelt by the default transmit queue selection
>>>>> implementation which tries to evenly distribute the traffic across
>>>>> all available queues. This selection presumes that the queues are
>>>>> using an equal priority, however, the queues 1 and 2 are actually of
>>>>> higher priority (the classification of the queues is enabled in
>>fec_enet_enable_ring).
>>>>>
>>>>> This can lead to net scheduler warnings and continuous TX ring dumps
>>>>> when exercising the system with iperf.
>>>>>
>>>>> Use only queue 0 for all common traffic (no VLAN and P802.1p
>>>>> priority
>>>>> 0 and 1) and route level 2-7 through queue 1 and 2.
>>>>>
>>>>> Signed-off-by: Fugang Duan
>>>>> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
>>>>
>>>>If the queues are used for prioritization, and it does not have
>>>>multiple normal priority level queues, multiqueue is not what the
>>>>driver should have implemented.
>>> Firstly, HW multiple queues support:
>>> - Traffic-shaping bandwidth distribution supports credit-based and
>>> round-robin-based policies. Either policy can be combined with
>>> time-based shaping.
>>> - AVB (Audio Video Bridging, IEEE 802.1Qav) features:
>>> * Credit-based bandwidth distribution policy can be combined
>>with
>>> time-based shaping
>>> * AVB endpoint talker and listener support
>>> * Support for arbitration between different priority traffic
>>> (for
>>> example, AVB class A, AVB class B, and non-AVB) Round-robin-based
>>> policies:
>>> It has the same priority for three queues: In the round-robin QoS
>>> scheme, each queue is given an equal opportunity to transmit one
>>> frame. For example, if queue n has a frame to transmit, the queue
>>> transmits its frame. After queue n has transmitted its frame, or if
>>> queue n does not have a frame to transmit, queue n+1 is then allowed
>>> to transmit its frame, and so on.
>>>
>>> Credit-based policies:
>>> The AVB credit based shaper acts independently, per class, to control
>>> the bandwidth distribution between normal traffic and time-sensitive
>>> traffic with respect to the total link bandwidth available.
>>> Credit based shaper conbined with time-based shaping:
>>> - priority: ClassA queue > ClassB queue > best effort
>>> - ensure the queue bandwidth as user set based on time-
>>based shaping
>>> algorithms (transmitter transmit frame from three queue in turn based
>>> on time-based shaping algorithms)
>>> And in real AVB case, each streaming can be independent, and are
>>> fixed on related queue. Then driver level should implement
>>> .ndo_select_queue() to put the streaming into related queue. That is
>>> what the patch did.
>>>
>>> The current driver config the three queue to credit-based policies
>>> (AVB), the patch seems no problem for the implementation. Do you have
>>> any suggestion ?
>>>
>>
>>I tried using the round robin mode by adding this:
>>
>>+ /* Set Round-Robin policy */
>>+ writel(1, fep->hwp + FEC_QOS_SCHEME);
>>
>>After a while I got the warning non the less:
>>
>>WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>>dev_watchdog+0x248/0x24c NETDEV WATCHDOG: eth0
RE: [PATCH] net: fec: select queue depending on VLAN priority
From: Stefan Agner Sent: Thursday, May 11, 2017 12:08 PM
>To: Andy Duan
>Cc: David Miller ; and...@lunn.ch;
>feste...@gmail.com; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>
>On 2017-05-09 19:42, Andy Duan wrote:
>> From: David Miller Sent: Tuesday, May 09, 2017
>> 9:39 PM
>>>To: ste...@agner.ch
>>>Cc: Andy Duan ; and...@lunn.ch;
>>>feste...@gmail.com; net...@vger.kernel.org; linux-
>>>ker...@vger.kernel.org
>>>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>>>
>>>From: Stefan Agner
>>>Date: Mon, 8 May 2017 22:37:08 -0700
>>>
>>>> Since the addition of the multi queue code with commit 59d0f7465644
>>>> ("net: fec: init multi queue date structure") the queue selection
>>>> has been handelt by the default transmit queue selection
>>>> implementation which tries to evenly distribute the traffic across
>>>> all available queues. This selection presumes that the queues are
>>>> using an equal priority, however, the queues 1 and 2 are actually of
>>>> higher priority (the classification of the queues is enabled in
>fec_enet_enable_ring).
>>>>
>>>> This can lead to net scheduler warnings and continuous TX ring dumps
>>>> when exercising the system with iperf.
>>>>
>>>> Use only queue 0 for all common traffic (no VLAN and P802.1p
>>>> priority
>>>> 0 and 1) and route level 2-7 through queue 1 and 2.
>>>>
>>>> Signed-off-by: Fugang Duan
>>>> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
>>>
>>>If the queues are used for prioritization, and it does not have
>>>multiple normal priority level queues, multiqueue is not what the
>>>driver should have implemented.
>> Firstly, HW multiple queues support:
>> - Traffic-shaping bandwidth distribution supports credit-based and
>> round-robin-based policies. Either policy can be combined with
>> time-based shaping.
>> - AVB (Audio Video Bridging, IEEE 802.1Qav) features:
>> * Credit-based bandwidth distribution policy can be combined
>with
>> time-based shaping
>> * AVB endpoint talker and listener support
>> * Support for arbitration between different priority traffic
>> (for
>> example, AVB class A, AVB class B, and non-AVB) Round-robin-based
>> policies:
>> It has the same priority for three queues: In the round-robin QoS
>> scheme, each queue is given an equal opportunity to transmit one
>> frame. For example, if queue n has a frame to transmit, the queue
>> transmits its frame. After queue n has transmitted its frame, or if
>> queue n does not have a frame to transmit, queue n+1 is then allowed
>> to transmit its frame, and so on.
>>
>> Credit-based policies:
>> The AVB credit based shaper acts independently, per class, to control
>> the bandwidth distribution between normal traffic and time-sensitive
>> traffic with respect to the total link bandwidth available.
>> Credit based shaper conbined with time-based shaping:
>> - priority: ClassA queue > ClassB queue > best effort
>> - ensure the queue bandwidth as user set based on time-
>based shaping
>> algorithms (transmitter transmit frame from three queue in turn based
>> on time-based shaping algorithms)
>> And in real AVB case, each streaming can be independent, and are
>> fixed on related queue. Then driver level should implement
>> .ndo_select_queue() to put the streaming into related queue. That is
>> what the patch did.
>>
>> The current driver config the three queue to credit-based policies
>> (AVB), the patch seems no problem for the implementation. Do you have
>> any suggestion ?
>>
>
>I tried using the round robin mode by adding this:
>
>+ /* Set Round-Robin policy */
>+ writel(1, fep->hwp + FEC_QOS_SCHEME);
>
>After a while I got the warning non the less:
>
>WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>dev_watchdog+0x248/0x24c NETDEV WATCHDOG: eth0 (fec): transmit queue
>2 timed out Modules linked in:
>CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.11.0-rc1-00058-g56d22eced8bc-
>dirty #377 Hardware name: Freescale i.MX7 Dual (Device Tree) []
>(unwind_backtrace) from [] (show_stack+0x10/0x14) []
>(show_stack) from [] (dump_stack+0x78/0x8c) []
>(dump_stack) from [] (__warn+0xe8/0x100) []
>(__warn) from [] (warn_sl
Re: [PATCH] net: fec: select queue depending on VLAN priority
On 2017-05-09 19:42, Andy Duan wrote:
> From: David Miller Sent: Tuesday, May 09, 2017 9:39 PM
>>To: ste...@agner.ch
>>Cc: Andy Duan ; and...@lunn.ch;
>>feste...@gmail.com; net...@vger.kernel.org; linux-
>>ker...@vger.kernel.org
>>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>>
>>From: Stefan Agner
>>Date: Mon, 8 May 2017 22:37:08 -0700
>>
>>> Since the addition of the multi queue code with commit 59d0f7465644
>>> ("net: fec: init multi queue date structure") the queue selection has
>>> been handelt by the default transmit queue selection implementation
>>> which tries to evenly distribute the traffic across all available
>>> queues. This selection presumes that the queues are using an equal
>>> priority, however, the queues 1 and 2 are actually of higher priority
>>> (the classification of the queues is enabled in fec_enet_enable_ring).
>>>
>>> This can lead to net scheduler warnings and continuous TX ring dumps
>>> when exercising the system with iperf.
>>>
>>> Use only queue 0 for all common traffic (no VLAN and P802.1p priority
>>> 0 and 1) and route level 2-7 through queue 1 and 2.
>>>
>>> Signed-off-by: Fugang Duan
>>> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
>>
>>If the queues are used for prioritization, and it does not have multiple
>>normal
>>priority level queues, multiqueue is not what the driver should have
>>implemented.
> Firstly, HW multiple queues support:
> - Traffic-shaping bandwidth distribution supports credit-based and
> round-robin-based policies. Either policy can be combined with
> time-based shaping.
> - AVB (Audio Video Bridging, IEEE 802.1Qav) features:
> * Credit-based bandwidth distribution policy can be combined
> with
> time-based shaping
> * AVB endpoint talker and listener support
> * Support for arbitration between different priority traffic
> (for
> example, AVB class A, AVB class B, and non-AVB)
> Round-robin-based policies:
> It has the same priority for three queues: In the round-robin QoS
> scheme, each queue is given an equal opportunity to transmit one
> frame. For example, if queue n has a frame to transmit, the queue
> transmits its frame. After queue n has transmitted its frame, or if
> queue n does not have a frame to transmit, queue n+1 is then allowed
> to transmit its frame, and so on.
>
> Credit-based policies:
> The AVB credit based shaper acts independently, per class, to control
> the bandwidth distribution between normal traffic and time-sensitive
> traffic with respect to the total link bandwidth available.
> Credit based shaper conbined with time-based shaping:
> - priority: ClassA queue > ClassB queue > best effort
> - ensure the queue bandwidth as user set based on time-based
> shaping
> algorithms (transmitter transmit frame from three queue in turn based
> on time-based shaping algorithms)
> And in real AVB case, each streaming can be independent, and are
> fixed on related queue. Then driver level should implement
> .ndo_select_queue() to put the streaming into related queue. That is
> what the patch did.
>
> The current driver config the three queue to credit-based policies
> (AVB), the patch seems no problem for the implementation. Do you have
> any suggestion ?
>
I tried using the round robin mode by adding this:
+ /* Set Round-Robin policy */
+ writel(1, fep->hwp + FEC_QOS_SCHEME);
After a while I got the warning non the less:
WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
dev_watchdog+0x248/0x24c
NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted
4.11.0-rc1-00058-g56d22eced8bc-dirty #377
Hardware name: Freescale i.MX7 Dual (Device Tree)
[] (unwind_backtrace) from [] (show_stack+0x10/0x14)
[] (show_stack) from [] (dump_stack+0x78/0x8c)
[] (dump_stack) from [] (__warn+0xe8/0x100)
[] (__warn) from [] (warn_slowpath_fmt+0x38/0x48)
[] (warn_slowpath_fmt) from []
(dev_watchdog+0x248/0x24c)
[] (dev_watchdog) from [] (call_timer_fn+0x28/0x98)
[] (call_timer_fn) from [] (expire_timers+0xa0/0xac)
[] (expire_timers) from []
(run_timer_softirq+0x9c/0x194)
[] (run_timer_softirq) from []
(__do_softirq+0x114/0x234)
[] (__do_softirq) from [] (irq_exit+0xcc/0x108)
[] (irq_exit) from []
(__handle_domain_irq+0x80/0xec)
[] (__handle_domain_irq) from []
(gic_handle_irq+0x48/0x8c)
[] (gic_handle_irq) from [] (__irq_svc+0x58/0x8c)
Exception stack(0xc1001f28 to 0xc1001f70)
1f20: 0001 c022fdc0
Re: [PATCH] net: fec: select queue depending on VLAN priority
On 2017-05-09 06:39, David Miller wrote:
> From: Stefan Agner
> Date: Mon, 8 May 2017 22:37:08 -0700
>
>> Since the addition of the multi queue code with commit 59d0f7465644
>> ("net: fec: init multi queue date structure") the queue selection
>> has been handelt by the default transmit queue selection
>> implementation which tries to evenly distribute the traffic across
>> all available queues. This selection presumes that the queues are
>> using an equal priority, however, the queues 1 and 2 are actually
>> of higher priority (the classification of the queues is enabled in
>> fec_enet_enable_ring).
>>
>> This can lead to net scheduler warnings and continuous TX ring
>> dumps when exercising the system with iperf.
>>
>> Use only queue 0 for all common traffic (no VLAN and P802.1p
>> priority 0 and 1) and route level 2-7 through queue 1 and 2.
>>
>> Signed-off-by: Fugang Duan
>> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
>
> If the queues are used for prioritization, and it does not have
> multiple normal priority level queues, multiqueue is not what the
> driver should have implemented.
As Andy mentioned, there is also a round-robin mode. I'll try that.
What would be the proper way to use the prioritized queues?
--
Stefan
RE: [PATCH] net: fec: select queue depending on VLAN priority
From: David Miller Sent: Tuesday, May 09, 2017 9:39 PM
>To: ste...@agner.ch
>Cc: Andy Duan ; and...@lunn.ch;
>feste...@gmail.com; net...@vger.kernel.org; linux-
>ker...@vger.kernel.org
>Subject: Re: [PATCH] net: fec: select queue depending on VLAN priority
>
>From: Stefan Agner
>Date: Mon, 8 May 2017 22:37:08 -0700
>
>> Since the addition of the multi queue code with commit 59d0f7465644
>> ("net: fec: init multi queue date structure") the queue selection has
>> been handelt by the default transmit queue selection implementation
>> which tries to evenly distribute the traffic across all available
>> queues. This selection presumes that the queues are using an equal
>> priority, however, the queues 1 and 2 are actually of higher priority
>> (the classification of the queues is enabled in fec_enet_enable_ring).
>>
>> This can lead to net scheduler warnings and continuous TX ring dumps
>> when exercising the system with iperf.
>>
>> Use only queue 0 for all common traffic (no VLAN and P802.1p priority
>> 0 and 1) and route level 2-7 through queue 1 and 2.
>>
>> Signed-off-by: Fugang Duan
>> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
>
>If the queues are used for prioritization, and it does not have multiple normal
>priority level queues, multiqueue is not what the driver should have
>implemented.
Firstly, HW multiple queues support:
- Traffic-shaping bandwidth distribution supports credit-based and
round-robin-based policies. Either policy can be combined with time-based
shaping.
- AVB (Audio Video Bridging, IEEE 802.1Qav) features:
* Credit-based bandwidth distribution policy can be combined
with time-based shaping
* AVB endpoint talker and listener support
* Support for arbitration between different priority traffic
(for example, AVB class A, AVB class B, and non-AVB)
Round-robin-based policies:
It has the same priority for three queues: In the round-robin QoS
scheme, each queue is given an equal opportunity to transmit one frame. For
example, if queue n has a frame to transmit, the queue transmits its frame.
After queue n has transmitted its frame, or if queue n does not have a frame to
transmit, queue n+1 is then allowed to transmit its frame, and so on.
Credit-based policies:
The AVB credit based shaper acts independently, per class, to control
the bandwidth distribution between normal traffic and time-sensitive traffic
with respect to the total link bandwidth available.
Credit based shaper conbined with time-based shaping:
- priority: ClassA queue > ClassB queue > best effort
- ensure the queue bandwidth as user set based on time-based
shaping algorithms (transmitter transmit frame from three queue in turn based
on time-based shaping algorithms)
And in real AVB case, each streaming can be independent, and are fixed
on related queue. Then driver level should implement .ndo_select_queue() to put
the streaming into related queue. That is what the patch did.
The current driver config the three queue to credit-based policies (AVB), the
patch seems no problem for the implementation. Do you have any suggestion ?
Andy
Re: [PATCH] net: fec: select queue depending on VLAN priority
From: Stefan Agner
Date: Mon, 8 May 2017 22:37:08 -0700
> Since the addition of the multi queue code with commit 59d0f7465644
> ("net: fec: init multi queue date structure") the queue selection
> has been handelt by the default transmit queue selection
> implementation which tries to evenly distribute the traffic across
> all available queues. This selection presumes that the queues are
> using an equal priority, however, the queues 1 and 2 are actually
> of higher priority (the classification of the queues is enabled in
> fec_enet_enable_ring).
>
> This can lead to net scheduler warnings and continuous TX ring
> dumps when exercising the system with iperf.
>
> Use only queue 0 for all common traffic (no VLAN and P802.1p
> priority 0 and 1) and route level 2-7 through queue 1 and 2.
>
> Signed-off-by: Fugang Duan
> Fixes: 59d0f7465644 ("net: fec: init multi queue date structure")
If the queues are used for prioritization, and it does not have
multiple normal priority level queues, multiqueue is not what the
driver should have implemented.
Re: [PATCH] net: fec: select queue depending on VLAN priority
On Mon, May 08, 2017 at 10:37:08PM -0700, Stefan Agner wrote:
> Since the addition of the multi queue code with commit 59d0f7465644
> ("net: fec: init multi queue date structure") the queue selection
> has been handelt by the default transmit queue selection
> implementation which tries to evenly distribute the traffic across
> all available queues. This selection presumes that the queues are
> using an equal priority, however, the queues 1 and 2 are actually
> of higher priority (the classification of the queues is enabled in
> fec_enet_enable_ring).
>
> This can lead to net scheduler warnings and continuous TX ring
> dumps when exercising the system with iperf.
>
> Use only queue 0 for all common traffic (no VLAN and P802.1p
> priority 0 and 1) and route level 2-7 through queue 1 and 2.
Hi Stefan
Did you try:
vconfig set_egress_map eth0.42 0 7
ip addr add 10.42.42.42/24 eth0.42
iperf -c 10.42.42.1
i.e. send a continuous stream on one of the higher priority queues.
>From what was said earlier in this thread, isn't queue 0 going to be
starved? As well as this patch, don't we also need some default
bandwidth allocations to the queues to ensure queue 0 does get some
bandwidth?
Andrew
