Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
On Wed, 2016-04-06 at 10:39 -0700, Dave Taht wrote: > > > The hope had been the original codel.h would have been reusable, > > > which is not the case at present. > > So what's the strategy for making it happen? > Strategy? to meander towards a result that gives low latency to all > stations, no matter their bandwidth, on several chipsets. I meant "strategy for making the code reusable". Or something like that anyway. I don't see the point in trying and then failing. Here we're adding a completely different version of codel to the kernel - why? What makes this version unusable for the original usage in include/net/codel.h? Can't we replace that one with the newer version and actually use the same file here? Or - why bother with the header file to make it shareable, if we're not even attempting to do that? johannes -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
On Wed, Apr 6, 2016 at 12:21 AM, Johannes Berg wrote: > [removing other lists since they spam me with moderation bounces] I have added your email address be accepted to the codel, make-wifi-fast lists. My apologies for the bounces. The people on those lists generally do not have the time to tackle the volume of traffic on linux-wireless. >> The hope had been the original codel.h would have been reusable, >> which is not the case at present. > > So what's the strategy for making it happen? Strategy? to meander towards a result that gives low latency to all stations, no matter their bandwidth, on several chipsets. The holy grail from my viewpoint is to get airtime fairness, better mac utilization, slow stations not starving fast ones, more stations servicable, and so on, and my focus has generally been on having an architecture that applied equally to APs and clients. Getting clients alone to have a queuing latency reduction of these orders of magnitude on uploads at low rates would be a huge win, but not the holy grail. It was really nice to have michal's proof of concept(s) show up and show fq_codel-like benefits at both low and high speeds on wifi, but it is clear more architectural rework is required to fit the theory into the reality. > Unless there is one, I > don't see the point in making the code more complicated than it already > has to be anyway. +1. Next steps were - get toke's and my testbeds up - avery/tim/myself to keep hammering at the ath9k - michal exploring dql - jonathon poking at it with cake-like ideas - and anyone else that cares to join in on finally fixing bufferbloat on wifi. and maybe put together a videoconference in 2-3 weeks or so with where we are stuck at (felix will be off vacation, too, I think). There are still multiple points where we all talk past each other. Me, for example, am overly fixated on having a per station queue to start with (which in the case of a client is two stations - one multicast/mgtmt frames and regular traffic) and not dealing with tids until much later in the process. Unfortunately it seems the hook is very late in the process. > > johannes -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Make-wifi-fast] [PATCHv2 1/2] mac80211: implement fair queuing per txq
> On 6 Apr, 2016, at 10:16, Michal Kazior wrote: > > When a driver asks mac80211 to dequeue given txq it implies a > destination station as well. This is important because 802.11 > aggregation can be performed only on groups of packets going to a > single station on a single tid. > > Cake - as I understand it - doesn't really *guarantee* maintaining > this. Keep in mind you can run with hundreds of stations connected. > > You don't really want to burden drivers with sorting this grouping up > themselves (and hence coerce them into introducing another level of > intermediate queues, bis). Well, no. Cake isn’t designed to maintain per-station queues explicitly, though it does have support for stochastic fairness between hosts. It is also blissfuly unaware of the requirements of wifi aggregation, largely because the standard qdisc interface is likewise ignorant. I’m therefore not suggesting that you use Cake as-is. What I’m pointing at instead is the set-associative hash, which could easily be tweaked to put greater emphasis on avoiding putting multiple stations’ traffic in one queue, while maintaining the performance benefits of a fixed queue pool indexed by a hash table, and an extended operating region in which flow isolation is maintained. You can then have a linked-list of queues assigned to a particular station, so that when a packet for a particular station is requested, you can easily locate one. I hadn’t appreciated, though, that the TXQ struct was station-specific. This wasn’t obvious from the code fragments posted, so it looked like packets that incurred hash collisions would be dumped into a single overflow queue. - Jonathan Morton -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
[removing other lists since they spam me with moderation bounces] > The hope had been the original codel.h would have been reusable, > which is not the case at present. So what's the strategy for making it happen? Unless there is one, I don't see the point in making the code more complicated than it already has to be anyway. johannes -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
On Wed, 2016-04-06 at 07:35 +0200, Michal Kazior wrote: > I just wanted to follow the suggested/implied usage of codel code and > keep modifications to a minimum. I could very well just assimilate it > if you wish. I don't really feel all that strongly about it, but I also don't see the point. It makes it harder to look for the code though, and that seems fairly pointless. Btw, just realized that there's also __u32 in there which you should probably remove and use just u32. Also don't #include > This follows net/sched/sch_fq_codel.h. I can put up a comment to > explain what it's supposed to do? > Ok, fair enough. johannes -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Make-wifi-fast] [PATCHv2 1/2] mac80211: implement fair queuing per txq
On 6 April 2016 at 08:03, Jonathan Morton wrote:
>
>> On 6 Apr, 2016, at 08:35, Michal Kazior wrote:
>>
>> Packets can be destined to different stations/txqs. At enqueue time I
>> do a partial hash of a packet to get an "index" which I then use to
>> address a txq_flow from per-radio list (out of 4096 of them). You can
>> end up with a situtation like this:
>> - packet A hashing to X destined to txq P which is VI
>> - packet B hashing to X destined to txq Q which is BK
>>
>> You can't use the same txq_flow for both A and B because you want to
>> maintain packets per txqs more than you want to maintain them per flow
>> (you don't want to queue BK traffic onto VI or vice versa as an
>> artifact, do you? ;). When a txq_flow doesn't have a txqi it is bound.
>> Later, if a collision happens (i.e. resulting txq_flow has non-NULL
>> txqi) the "embedded" per-txq flow is used:
>>
>> struct txq_info {
>> - struct sk_buff_head queue;
>> + struct txq_flow flow; // <--- this
>>
>> When txq_flow becomes empty its txqi is reset.
>>
>> The embedded flow is otherwise treated like any other flow, i.e. it
>> can be linked to old_flows and new_flows.
>
> This smells like a very fragile and complex solution to the collision
> problem. You may want to look at how Cake solves it.
>
> I use a separate pool of flows per traffic class (essentially, VO/VI/BE/BK),
> and there is also a set-associative hash to take care of the birthday
> problem. The latter has an order-of-magnitude effect on the general flow
> collision rate once you get into the tens of flows, for very little CPU cost.
When a driver asks mac80211 to dequeue given txq it implies a
destination station as well. This is important because 802.11
aggregation can be performed only on groups of packets going to a
single station on a single tid.
Cake - as I understand it - doesn't really *guarantee* maintaining
this. Keep in mind you can run with hundreds of stations connected.
You don't really want to burden drivers with sorting this grouping up
themselves (and hence coerce them into introducing another level of
intermediate queues, bis).
Without the per-txq fallback flow (regardless of using fq_codel-like
scheme or cake-like scheme in mac80211) you'll need to modify
codel_dequeue() itself to compensate and re-queue/skip frames not
belonging to requested txq.
I'm not sure it's worth it for initial fair-queuing implementation.
Michał
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [Make-wifi-fast] [PATCHv2 1/2] mac80211: implement fair queuing per txq
> On 6 Apr, 2016, at 08:35, Michal Kazior wrote:
>
> Packets can be destined to different stations/txqs. At enqueue time I
> do a partial hash of a packet to get an "index" which I then use to
> address a txq_flow from per-radio list (out of 4096 of them). You can
> end up with a situtation like this:
> - packet A hashing to X destined to txq P which is VI
> - packet B hashing to X destined to txq Q which is BK
>
> You can't use the same txq_flow for both A and B because you want to
> maintain packets per txqs more than you want to maintain them per flow
> (you don't want to queue BK traffic onto VI or vice versa as an
> artifact, do you? ;). When a txq_flow doesn't have a txqi it is bound.
> Later, if a collision happens (i.e. resulting txq_flow has non-NULL
> txqi) the "embedded" per-txq flow is used:
>
> struct txq_info {
> - struct sk_buff_head queue;
> + struct txq_flow flow; // <--- this
>
> When txq_flow becomes empty its txqi is reset.
>
> The embedded flow is otherwise treated like any other flow, i.e. it
> can be linked to old_flows and new_flows.
This smells like a very fragile and complex solution to the collision problem.
You may want to look at how Cake solves it.
I use a separate pool of flows per traffic class (essentially, VO/VI/BE/BK),
and there is also a set-associative hash to take care of the birthday problem.
The latter has an order-of-magnitude effect on the general flow collision rate
once you get into the tens of flows, for very little CPU cost.
- Jonathan Morton
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
On 5 April 2016 at 15:57, Johannes Berg wrote:
> On Thu, 2016-03-31 at 12:28 +0200, Michal Kazior wrote:
>
>> +++ b/net/mac80211/codel.h
>> +++ b/net/mac80211/codel_i.h
>
> Do we really need all this code in .h files? It seems very odd to me to
> have all the algorithm implementation there rather than a C file, you
> should (can?) only include codel.h into a single C file anyway.
I just wanted to follow the suggested/implied usage of codel code and
keep modifications to a minimum. I could very well just assimilate it
if you wish.
>> struct txq_info {
>> - struct sk_buff_head queue;
>> + struct txq_flow flow;
>> + struct list_head new_flows;
>> + struct list_head old_flows;
>
> This is confusing, can you please document that? Why are there two
> lists of flows, *and* an embedded flow? Is the embedded flow on any of
> the lists?
The new/old flows is follows the same principle as net/sched/sch_fq_codel.c
The embedded flow is for possible collisions, explained below.
Nevertheless I'll add more comments on what-is-what-and-why.
>> + u32 backlog_bytes;
>> + u32 backlog_packets;
>> + u32 drop_codel;
>
> Would it make some sense to at least conceptually layer this a bit?
> I.e. rather than calling this "drop_codel" call it "drop_congestion" or
> something like that?
Sure, I'll change it.
>> +/**
>> + * struct txq_flow - per traffic flow queue
>> + *
>> + * This structure is used to distinguish and queue different traffic flows
>> + * separately for fair queueing/AQM purposes.
>> + *
>> + * @txqi: txq_info structure it is associated at given time
>
> Do we actually have to keep that? It's on a list per txqi, no?
It's used to track ownership.
Packets can be destined to different stations/txqs. At enqueue time I
do a partial hash of a packet to get an "index" which I then use to
address a txq_flow from per-radio list (out of 4096 of them). You can
end up with a situtation like this:
- packet A hashing to X destined to txq P which is VI
- packet B hashing to X destined to txq Q which is BK
You can't use the same txq_flow for both A and B because you want to
maintain packets per txqs more than you want to maintain them per flow
(you don't want to queue BK traffic onto VI or vice versa as an
artifact, do you? ;). When a txq_flow doesn't have a txqi it is bound.
Later, if a collision happens (i.e. resulting txq_flow has non-NULL
txqi) the "embedded" per-txq flow is used:
struct txq_info {
- struct sk_buff_head queue;
+ struct txq_flow flow; // <--- this
When txq_flow becomes empty its txqi is reset.
The embedded flow is otherwise treated like any other flow, i.e. it
can be linked to old_flows and new_flows.
>> + * @flowchain: can be linked to other flows for RR purposes
>
> RR?
Round-robin. Assuming it's correct to call fq_codel an RR scheme?
>> +void ieee80211_teardown_flows(struct ieee80211_local *local)
>> +{
>> + struct ieee80211_fq *fq = &local->fq;
>> + struct ieee80211_sub_if_data *sdata;
>> + struct sta_info *sta;
>> + int i;
>> +
>> + if (!local->ops->wake_tx_queue)
>> + return;
>> +
>> + list_for_each_entry_rcu(sta, &local->sta_list, list)
>> + for (i = 0; i < IEEE80211_NUM_TIDS; i++)
>> + ieee80211_purge_txq(local,
>> + to_txq_info(sta->sta.txq[i]));
>> +
>> + list_for_each_entry_rcu(sdata, &local->interfaces, list)
>> + ieee80211_purge_txq(local, to_txq_info(sdata->vif.txq));
>
> Using RCU iteration here seems rather strange, since it's a teardown
> flow? That doesn't seem necessary, since it's control path and must be
> holding appropriate locks anyway to make sure nothing is added to the
> lists.
You're probably right. I'll look into changing it.
>
>> + skb = codel_dequeue(flow,
>> + &flow->backlog,
>> + 0,
>> + &flow->cvars,
>> + &fq->cparams,
>> + codel_get_time(),
>> + false);
>
> What happened here? :)
I'm not a huge fan of wrapping functions with a lot of (ugly-looking)
arguments. I can make it a different ugly if you want :)
>> + if (!skb) {
>> + if ((head == &txqi->new_flows) &&
>> + !list_empty(&txqi->old_flows)) {
>> + list_move_tail(&flow->flowchain, &txqi->old_flows);
>> + } else {
>> + list_del_init(&flow->flowchain);
>> + flow->txqi = NULL;
>> + }
>> + goto begin;
>> + }
>
> Ouch. Any way you can make that easier to follow?
This follows net/sched/sch_fq_codel.h. I can put up a comment to
explain what it's supposed to do?
Michał
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
thx for the review!
On Tue, Apr 5, 2016 at 6:57 AM, Johannes Berg wrote:
> On Thu, 2016-03-31 at 12:28 +0200, Michal Kazior wrote:
>
>> +++ b/net/mac80211/codel.h
>> +++ b/net/mac80211/codel_i.h
>
> Do we really need all this code in .h files? It seems very odd to me to
> have all the algorithm implementation there rather than a C file, you
> should (can?) only include codel.h into a single C file anyway.
The hope had been the original codel.h would have been reusable, which
is not the case at present.
>
>> struct txq_info {
>> - struct sk_buff_head queue;
>> + struct txq_flow flow;
>> + struct list_head new_flows;
>> + struct list_head old_flows;
>
> This is confusing, can you please document that? Why are there two
> lists of flows, *and* an embedded flow? Is the embedded flow on any of
> the lists?
To explain the new and old flow concepts, there's
https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06 which is in the
ietf editors queue for final publication and doesn't have a final name
yet.
The embedded flow concept is michal's and I'm not convinced it's the
right idea as yet.
>
>> + u32 backlog_bytes;
>> + u32 backlog_packets;
>> + u32 drop_codel;
>
> Would it make some sense to at least conceptually layer this a bit?
> I.e. rather than calling this "drop_codel" call it "drop_congestion" or
> something like that?
Is there a more generic place overall in ieee80211 to record per-sta
backlogs, drops and marks?
>> + skb = codel_dequeue(flow,
>> + &flow->backlog,
>> + 0,
>> + &flow->cvars,
>> + &fq->cparams,
>> + codel_get_time(),
>> + false);
>
> What happened here? :)
Magic.
>
>> + if (!skb) {
>> + if ((head == &txqi->new_flows) &&
>> + !list_empty(&txqi->old_flows)) {
>> + list_move_tail(&flow->flowchain, &txqi->old_flows);
>> + } else {
>> + list_del_init(&flow->flowchain);
>> + flow->txqi = NULL;
>> + }
>> + goto begin;
>> + }
>
> Ouch. Any way you can make that easier to follow?
It made my brain hurt in the original code, too, but it is eric
optimizing out cycles at his finest.
if the the new_flows list is expired or done, switch to the old_flows
list, if the old_flows list is done, go try selecting another queue to
pull from (which may or may not exist). see the pending rfc for a more
elongated version.
>
> johannes
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCHv2 1/2] mac80211: implement fair queuing per txq
On Thu, 2016-03-31 at 12:28 +0200, Michal Kazior wrote:
> +++ b/net/mac80211/codel.h
> +++ b/net/mac80211/codel_i.h
Do we really need all this code in .h files? It seems very odd to me to
have all the algorithm implementation there rather than a C file, you
should (can?) only include codel.h into a single C file anyway.
> struct txq_info {
> - struct sk_buff_head queue;
> + struct txq_flow flow;
> + struct list_head new_flows;
> + struct list_head old_flows;
This is confusing, can you please document that? Why are there two
lists of flows, *and* an embedded flow? Is the embedded flow on any of
the lists?
> + u32 backlog_bytes;
> + u32 backlog_packets;
> + u32 drop_codel;
Would it make some sense to at least conceptually layer this a bit?
I.e. rather than calling this "drop_codel" call it "drop_congestion" or
something like that?
> @@ -977,12 +978,9 @@ static void ieee80211_do_stop(struct
> ieee80211_sub_if_data *sdata,
> if (sdata->vif.txq) {
> struct txq_info *txqi = to_txq_info(sdata->vif.txq);
>
> - spin_lock_bh(&txqi->queue.lock);
> - ieee80211_purge_tx_queue(&local->hw, &txqi->queue);
> - txqi->byte_cnt = 0;
> - spin_unlock_bh(&txqi->queue.lock);
> -
> - atomic_set(&sdata->txqs_len[txqi->txq.ac], 0);
> + spin_lock_bh(&fq->lock);
> + ieee80211_purge_txq(local, txqi);
> + spin_unlock_bh(&fq->lock);
This isn't very nice - you're going from locking a single txqi to
having a global hardware lock.
It's probably fine in this particular case, but I'll need to look for
other places :)
> +/**
> + * struct txq_flow - per traffic flow queue
> + *
> + * This structure is used to distinguish and queue different traffic flows
> + * separately for fair queueing/AQM purposes.
> + *
> + * @txqi: txq_info structure it is associated at given time
Do we actually have to keep that? It's on a list per txqi, no?
> + * @flowchain: can be linked to other flows for RR purposes
RR?
> +void ieee80211_teardown_flows(struct ieee80211_local *local)
> +{
> + struct ieee80211_fq *fq = &local->fq;
> + struct ieee80211_sub_if_data *sdata;
> + struct sta_info *sta;
> + int i;
> +
> + if (!local->ops->wake_tx_queue)
> + return;
> +
> + list_for_each_entry_rcu(sta, &local->sta_list, list)
> + for (i = 0; i < IEEE80211_NUM_TIDS; i++)
> + ieee80211_purge_txq(local,
> + to_txq_info(sta->sta.txq[i]));
> +
> + list_for_each_entry_rcu(sdata, &local->interfaces, list)
> + ieee80211_purge_txq(local, to_txq_info(sdata->vif.txq));
Using RCU iteration here seems rather strange, since it's a teardown
flow? That doesn't seem necessary, since it's control path and must be
holding appropriate locks anyway to make sure nothing is added to the
lists.
> + skb = codel_dequeue(flow,
> + &flow->backlog,
> + 0,
> + &flow->cvars,
> + &fq->cparams,
> + codel_get_time(),
> + false);
What happened here? :)
> + if (!skb) {
> + if ((head == &txqi->new_flows) &&
> + !list_empty(&txqi->old_flows)) {
> + list_move_tail(&flow->flowchain, &txqi->old_flows);
> + } else {
> + list_del_init(&flow->flowchain);
> + flow->txqi = NULL;
> + }
> + goto begin;
> + }
Ouch. Any way you can make that easier to follow?
johannes
--
To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCHv2 1/2] mac80211: implement fair queuing per txq
Qdiscs assume all packets regardless of
destination address are treated equally by
underlying an device link.
This isn't true for wireless where each node is a
link in it's own right with different and varying
signal quality over time.
Existing wireless behavior stuffs device tx queues
with no regard to link conditions. This can result
in queue buildup for slow stations and an inertia
worth of seconds making it impossible for small
bursty traffic to come through.
The current high-level idea is to keep roughly 1-2
txops worth of data in device tx queues to allow
short bursts to be handled responsively.
mac80211's software queues were designed to work
very closely with device tx queues. They are
required to make use of 802.11 packet aggregation
easily and efficiently.
However the logic imposed a per-AC queue limit.
With the limit too small mac80211 wasn't be able
to guarantee fairness across TIDs nor stations
because single burst to a slow station could
monopolize queues and reach per-AC limit
preventing traffic from other stations being
queued into mac80211's software queues. Having the
limit too large would make smart qdiscs, e.g.
fq_codel, a lot less efficient as they are
designed on the premise that they are very close
to the actualy device tx queues.
The patch implements fq_codel-ish logic in
mac80211's software queuing. This doesn't directly
translate to immediate and significant gains.
Moreover only wake_tx_queue based drivers will be
able to reap the benefits of fair queuing for now.
More work is required to make sure drivers keep
their device tx queues at minimum fill (instead of
clogging them up until they're full regardless of
link conditions). Only then the full effect of
fair queuing will be observable.
Signed-off-by: Michal Kazior
---
Notes:
v2:
* fix invalid ptr deref (I accidentally removed `info` ptr assignment..)
v1:
* move txq_limit and txq_cparams from ieee80211_hw to ieee80211_fq
* remove printks
* improve commit log
* various cleanups
* extra stats
* split out the core txq fairness changes
* should_drop() doesn't consider bursts
* codel target is hardcoded to 20ms
RFC v2:
* actually re-use txq_flows on enqueue [Felix]
* tune should_drop() to consider bursts wrt station expected tput
[Dave/Bob]
* make codel target time scale via ewma of estimated txqi service period
[Dave]
* generic tx scheduling (with time-based queue limit and naive hysteresis)
* tracking per-frame expected duration
* tracking per-txqi in-flight data duration
* tracking per-hw in-flight data duration
? in-flight means scheduled to driver and assumes driver does report
tx-status on actual tx-completion
* added a few debugfs entries
include/net/mac80211.h | 21 ++-
net/mac80211/agg-tx.c | 8 +-
net/mac80211/codel.h | 264 +++
net/mac80211/codel_i.h | 89 +++
net/mac80211/ieee80211_i.h | 45 +-
net/mac80211/iface.c | 24 ++-
net/mac80211/main.c| 9 +-
net/mac80211/rx.c | 2 +-
net/mac80211/sta_info.c| 10 +-
net/mac80211/sta_info.h| 27
net/mac80211/tx.c | 377 -
net/mac80211/util.c| 20 ++-
12 files changed, 817 insertions(+), 79 deletions(-)
create mode 100644 net/mac80211/codel.h
create mode 100644 net/mac80211/codel_i.h
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index a5cb1528..0ee51dbb361b 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -888,8 +888,18 @@ struct ieee80211_tx_info {
/* only needed before rate control */
unsigned long jiffies;
};
- /* NB: vif can be NULL for injected frames */
- struct ieee80211_vif *vif;
+ union {
+ /* NB: vif can be NULL for injected frames */
+ struct ieee80211_vif *vif;
+
+ /* When packets are enqueued on txq it's easy
+* to re-construct the vif pointer. There's no
+* more space in tx_info so it can be used to
+* store the necessary enqueue time for packet
+* sojourn time computation.
+*/
+ u64 enqueue_time;
+ };
struct ieee80211_key_conf *hw_key;
u32 flags;
/* 4 bytes free */
@@ -2113,9 +2123,6 @@ enum ieee80211_hw_flags {
* @n_cipher_schemes: a size of an array of cipher schemes definitions.
* @cipher_schemes: a pointer to an array of cipher scheme definitions
* supported by HW.
- *
- * @txq_ac_max_
